2024-09-16T07:01:53.302 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T07:01:53.314 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T07:01:53.391 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906617 branch: reef description: rados/standalone/{supported-random-distro$/{ubuntu_20.04} workloads/erasure-code} email: ceph-qa@ceph.com first_in_suite: false job_id: '7906617' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: reef ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 workunit: branch: reef sha1: ae7f37330c1a72493684eb1fcaed6f2a01195bb6 owner: scheduled_teuthology@teuthology priority: 931 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1390 sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 sleep_before_teardown: 0 subset: 5319/100000 suite: rados suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ae7f37330c1a72493684eb1fcaed6f2a01195bb6 targets: smithi162.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - erasure-code teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_22:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T07:01:53.391 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa; will attempt to use it 2024-09-16T07:01:53.392 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa/tasks 2024-09-16T07:01:53.392 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T07:01:53.394 INFO:teuthology.task.internal:Checking packages... 2024-09-16T07:01:53.415 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8' 2024-09-16T07:01:53.415 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T07:01:53.415 INFO:teuthology.packaging:ref: None 2024-09-16T07:01:53.415 INFO:teuthology.packaging:tag: None 2024-09-16T07:01:53.416 INFO:teuthology.packaging:branch: reef 2024-09-16T07:01:53.416 INFO:teuthology.packaging:sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T07:01:53.416 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=reef 2024-09-16T07:01:53.634 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-853-gae7f3733-1focal 2024-09-16T07:01:53.636 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T07:01:53.642 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T07:01:53.642 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T07:01:53.692 INFO:teuthology.task.internal:Saving configuration 2024-09-16T07:01:53.701 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T07:01:53.726 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T07:01:53.751 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906617', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 06:55:49.282090', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-16T07:01:53.751 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T07:01:53.758 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-16T07:01:53.758 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T07:01:53.892 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3a5d077370>, signals=[15]) 2024-09-16T07:01:53.893 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T07:01:53.898 INFO:teuthology.task.internal:Opening connections... 2024-09-16T07:01:53.898 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi162.front.sepia.ceph.com 2024-09-16T07:01:53.899 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:01:53.997 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T07:01:54.004 DEBUG:teuthology.orchestra.run.smithi162:> uname -m 2024-09-16T07:01:54.123 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2024-09-16T07:01:54.123 DEBUG:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2024-09-16T07:01:54.128 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="Ubuntu" 2024-09-16T07:01:54.129 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-16T07:01:54.129 INFO:teuthology.orchestra.run.smithi162.stdout:ID=ubuntu 2024-09-16T07:01:54.129 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE=debian 2024-09-16T07:01:54.129 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-16T07:01:54.129 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="20.04" 2024-09-16T07:01:54.129 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T07:01:54.129 INFO:teuthology.orchestra.run.smithi162.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T07:01:54.129 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T07:01:54.129 INFO:teuthology.orchestra.run.smithi162.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T07:01:54.129 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_CODENAME=focal 2024-09-16T07:01:54.129 INFO:teuthology.orchestra.run.smithi162.stdout:UBUNTU_CODENAME=focal 2024-09-16T07:01:54.130 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2024-09-16T07:01:54.154 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T07:01:54.162 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T07:01:54.200 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T07:01:54.200 DEBUG:teuthology.orchestra.run.smithi162:> test '!' -e /home/ubuntu/cephtest 2024-09-16T07:01:54.206 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T07:01:54.237 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T07:01:54.237 DEBUG:teuthology.orchestra.run.smithi162:> test -z $(ls -A /var/lib/ceph) 2024-09-16T07:01:54.254 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T07:01:54.291 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T07:01:54.308 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T07:01:54.308 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T07:01:54.308 DEBUG:teuthology.orchestra.run.smithi162:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T07:01:54.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T07:01:54.314 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-09-16T07:01:54.363 INFO:teuthology.orchestra.run.smithi162.stdout:5.4.0-124-generic 2024-09-16T07:01:54.363 INFO:teuthology.task.kernel:Running kernel on smithi162: 5.4.0-124-generic 2024-09-16T07:01:54.363 DEBUG:teuthology.orchestra.run.smithi162:> sudo apt-get clean 2024-09-16T07:01:54.477 DEBUG:teuthology.orchestra.run.smithi162:> sudo apt-get update 2024-09-16T07:01:54.713 INFO:teuthology.orchestra.run.smithi162.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-16T07:01:54.762 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T07:01:54.828 INFO:teuthology.orchestra.run.smithi162.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-16T07:01:55.148 INFO:teuthology.orchestra.run.smithi162.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [812 kB] 2024-09-16T07:01:55.180 INFO:teuthology.orchestra.run.smithi162.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-16T07:01:55.248 INFO:teuthology.orchestra.run.smithi162.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,195 kB] 2024-09-16T07:01:55.293 INFO:teuthology.orchestra.run.smithi162.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [472 kB] 2024-09-16T07:01:55.298 INFO:teuthology.orchestra.run.smithi162.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-16T07:01:55.299 INFO:teuthology.orchestra.run.smithi162.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-16T07:01:55.299 INFO:teuthology.orchestra.run.smithi162.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,117 kB] 2024-09-16T07:01:55.329 INFO:teuthology.orchestra.run.smithi162.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [436 kB] 2024-09-16T07:01:55.332 INFO:teuthology.orchestra.run.smithi162.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-16T07:01:55.333 INFO:teuthology.orchestra.run.smithi162.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,011 kB] 2024-09-16T07:01:55.339 INFO:teuthology.orchestra.run.smithi162.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-16T07:01:55.344 INFO:teuthology.orchestra.run.smithi162.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-16T07:01:55.345 INFO:teuthology.orchestra.run.smithi162.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.2 kB] 2024-09-16T07:01:55.346 INFO:teuthology.orchestra.run.smithi162.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-16T07:01:55.346 INFO:teuthology.orchestra.run.smithi162.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-16T07:01:55.346 INFO:teuthology.orchestra.run.smithi162.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-16T07:01:55.346 INFO:teuthology.orchestra.run.smithi162.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-16T07:01:55.353 INFO:teuthology.orchestra.run.smithi162.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,030 kB] 2024-09-16T07:01:55.681 INFO:teuthology.orchestra.run.smithi162.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,563 kB] 2024-09-16T07:01:55.881 INFO:teuthology.orchestra.run.smithi162.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [550 kB] 2024-09-16T07:01:55.894 INFO:teuthology.orchestra.run.smithi162.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-16T07:01:55.895 INFO:teuthology.orchestra.run.smithi162.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-16T07:01:55.896 INFO:teuthology.orchestra.run.smithi162.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,236 kB] 2024-09-16T07:01:56.007 INFO:teuthology.orchestra.run.smithi162.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [453 kB] 2024-09-16T07:01:56.019 INFO:teuthology.orchestra.run.smithi162.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-16T07:01:56.020 INFO:teuthology.orchestra.run.smithi162.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,230 kB] 2024-09-16T07:01:56.045 INFO:teuthology.orchestra.run.smithi162.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-16T07:01:56.060 INFO:teuthology.orchestra.run.smithi162.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-16T07:01:56.066 INFO:teuthology.orchestra.run.smithi162.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.0 kB] 2024-09-16T07:01:56.089 INFO:teuthology.orchestra.run.smithi162.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-16T07:01:56.090 INFO:teuthology.orchestra.run.smithi162.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-16T07:01:56.090 INFO:teuthology.orchestra.run.smithi162.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-16T07:01:56.128 INFO:teuthology.orchestra.run.smithi162.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-16T07:01:56.129 INFO:teuthology.orchestra.run.smithi162.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-16T07:01:56.130 INFO:teuthology.orchestra.run.smithi162.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-16T07:01:56.131 INFO:teuthology.orchestra.run.smithi162.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-16T07:01:56.210 INFO:teuthology.orchestra.run.smithi162.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-16T07:01:56.211 INFO:teuthology.orchestra.run.smithi162.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-16T07:01:56.212 INFO:teuthology.orchestra.run.smithi162.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-16T07:01:59.606 INFO:teuthology.orchestra.run.smithi162.stdout:Fetched 21.8 MB in 3s (6,961 kB/s) 2024-09-16T07:02:01.143 INFO:teuthology.orchestra.run.smithi162.stdout:Reading package lists... 2024-09-16T07:02:01.169 DEBUG:teuthology.orchestra.run.smithi162:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T07:02:01.244 INFO:teuthology.orchestra.run.smithi162.stdout:Reading package lists... 2024-09-16T07:02:01.454 INFO:teuthology.orchestra.run.smithi162.stdout:Building dependency tree... 2024-09-16T07:02:01.454 INFO:teuthology.orchestra.run.smithi162.stdout:Reading state information... 2024-09-16T07:02:01.655 INFO:teuthology.orchestra.run.smithi162.stdout:The following additional packages will be installed: 2024-09-16T07:02:01.655 INFO:teuthology.orchestra.run.smithi162.stdout: linux-generic linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-16T07:02:01.655 INFO:teuthology.orchestra.run.smithi162.stdout: linux-headers-generic linux-image-5.4.0-195-generic 2024-09-16T07:02:01.655 INFO:teuthology.orchestra.run.smithi162.stdout: linux-modules-5.4.0-195-generic linux-modules-extra-5.4.0-195-generic 2024-09-16T07:02:01.656 INFO:teuthology.orchestra.run.smithi162.stdout:Suggested packages: 2024-09-16T07:02:01.656 INFO:teuthology.orchestra.run.smithi162.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-16T07:02:01.656 INFO:teuthology.orchestra.run.smithi162.stdout:Recommended packages: 2024-09-16T07:02:01.656 INFO:teuthology.orchestra.run.smithi162.stdout: thermald 2024-09-16T07:02:01.712 INFO:teuthology.orchestra.run.smithi162.stdout:The following NEW packages will be installed: 2024-09-16T07:02:01.712 INFO:teuthology.orchestra.run.smithi162.stdout: linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-16T07:02:01.712 INFO:teuthology.orchestra.run.smithi162.stdout: linux-image-5.4.0-195-generic linux-modules-5.4.0-195-generic 2024-09-16T07:02:01.713 INFO:teuthology.orchestra.run.smithi162.stdout: linux-modules-extra-5.4.0-195-generic 2024-09-16T07:02:01.714 INFO:teuthology.orchestra.run.smithi162.stdout:The following packages will be upgraded: 2024-09-16T07:02:01.714 INFO:teuthology.orchestra.run.smithi162.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T07:02:01.897 INFO:teuthology.orchestra.run.smithi162.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-16T07:02:01.898 INFO:teuthology.orchestra.run.smithi162.stdout:Need to get 77.2 MB of archives. 2024-09-16T07:02:01.898 INFO:teuthology.orchestra.run.smithi162.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-16T07:02:01.898 INFO:teuthology.orchestra.run.smithi162.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-195-generic amd64 5.4.0-195.215 [15.0 MB] 2024-09-16T07:02:03.071 INFO:teuthology.orchestra.run.smithi162.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-195-generic amd64 5.4.0-195.215 [10.5 MB] 2024-09-16T07:02:03.348 INFO:teuthology.orchestra.run.smithi162.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-195-generic amd64 5.4.0-195.215 [39.2 MB] 2024-09-16T07:02:04.692 INFO:teuthology.orchestra.run.smithi162.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.195.193 [1,900 B] 2024-09-16T07:02:04.692 INFO:teuthology.orchestra.run.smithi162.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.195.193 [2,584 B] 2024-09-16T07:02:04.693 INFO:teuthology.orchestra.run.smithi162.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195 all 5.4.0-195.215 [11.0 MB] 2024-09-16T07:02:04.975 INFO:teuthology.orchestra.run.smithi162.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195-generic amd64 5.4.0-195.215 [1,386 kB] 2024-09-16T07:02:04.987 INFO:teuthology.orchestra.run.smithi162.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.195.193 [2,460 B] 2024-09-16T07:02:05.665 INFO:teuthology.orchestra.run.smithi162.stdout:Fetched 77.2 MB in 3s (23.6 MB/s) 2024-09-16T07:02:05.876 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package linux-modules-5.4.0-195-generic. 2024-09-16T07:02:07.560 INFO:teuthology.orchestra.run.smithi162.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-16T07:02:07.565 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../0-linux-modules-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T07:02:07.721 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:02:09.687 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package linux-image-5.4.0-195-generic. 2024-09-16T07:02:09.703 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../1-linux-image-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T07:02:09.791 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:02:10.898 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-195-generic. 2024-09-16T07:02:10.911 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T07:02:10.948 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:02:16.011 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../3-linux-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T07:02:16.138 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking linux-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T07:02:16.480 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T07:02:16.624 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking linux-image-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T07:02:16.961 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package linux-headers-5.4.0-195. 2024-09-16T07:02:16.975 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../5-linux-headers-5.4.0-195_5.4.0-195.215_all.deb ... 2024-09-16T07:02:17.010 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-16T07:02:23.160 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package linux-headers-5.4.0-195-generic. 2024-09-16T07:02:23.183 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../6-linux-headers-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T07:02:23.226 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:02:26.015 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T07:02:26.860 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking linux-headers-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T07:02:27.219 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:02:28.465 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:02:29.633 INFO:teuthology.orchestra.run.smithi162.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-195-generic 2024-09-16T07:02:29.633 INFO:teuthology.orchestra.run.smithi162.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-195-generic 2024-09-16T07:02:29.749 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:02:30.959 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-16T07:02:31.085 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up linux-image-generic (5.4.0.195.193) ... 2024-09-16T07:02:31.236 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:02:31.362 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up linux-headers-generic (5.4.0.195.193) ... 2024-09-16T07:02:31.488 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up linux-generic (5.4.0.195.193) ... 2024-09-16T07:02:31.631 INFO:teuthology.orchestra.run.smithi162.stdout:Processing triggers for linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:02:31.705 INFO:teuthology.orchestra.run.smithi162.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T07:02:31.705 INFO:teuthology.orchestra.run.smithi162.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-195-generic 2024-09-16T07:02:53.943 INFO:teuthology.orchestra.run.smithi162.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T07:02:53.943 INFO:teuthology.orchestra.run.smithi162.stdout:Sourcing file `/etc/default/grub' 2024-09-16T07:02:53.953 INFO:teuthology.orchestra.run.smithi162.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T07:02:53.963 INFO:teuthology.orchestra.run.smithi162.stdout:Generating grub configuration file ... 2024-09-16T07:02:54.328 INFO:teuthology.orchestra.run.smithi162.stdout:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-16T07:02:54.347 INFO:teuthology.orchestra.run.smithi162.stdout:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-16T07:02:54.783 INFO:teuthology.orchestra.run.smithi162.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-16T07:02:54.795 INFO:teuthology.orchestra.run.smithi162.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-16T07:02:56.921 INFO:teuthology.orchestra.run.smithi162.stdout:done 2024-09-16T07:02:57.363 DEBUG:teuthology.orchestra.run.smithi162:> dpkg -s linux-image-generic 2024-09-16T07:02:57.412 INFO:teuthology.orchestra.run.smithi162.stdout:Package: linux-image-generic 2024-09-16T07:02:57.412 INFO:teuthology.orchestra.run.smithi162.stdout:Status: install ok installed 2024-09-16T07:02:57.412 INFO:teuthology.orchestra.run.smithi162.stdout:Priority: optional 2024-09-16T07:02:57.412 INFO:teuthology.orchestra.run.smithi162.stdout:Section: kernel 2024-09-16T07:02:57.413 INFO:teuthology.orchestra.run.smithi162.stdout:Installed-Size: 20 2024-09-16T07:02:57.413 INFO:teuthology.orchestra.run.smithi162.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T07:02:57.413 INFO:teuthology.orchestra.run.smithi162.stdout:Architecture: amd64 2024-09-16T07:02:57.414 INFO:teuthology.orchestra.run.smithi162.stdout:Source: linux-meta 2024-09-16T07:02:57.414 INFO:teuthology.orchestra.run.smithi162.stdout:Version: 5.4.0.195.193 2024-09-16T07:02:57.414 INFO:teuthology.orchestra.run.smithi162.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-16T07:02:57.414 INFO:teuthology.orchestra.run.smithi162.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T07:02:57.414 INFO:teuthology.orchestra.run.smithi162.stdout:Recommends: thermald 2024-09-16T07:02:57.415 INFO:teuthology.orchestra.run.smithi162.stdout:Description: Generic Linux kernel image 2024-09-16T07:02:57.415 INFO:teuthology.orchestra.run.smithi162.stdout: This package will always depend on the latest generic kernel image 2024-09-16T07:02:57.416 INFO:teuthology.orchestra.run.smithi162.stdout: available. 2024-09-16T07:02:57.416 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-195-generic 2024-09-16T07:02:57.416 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T07:02:57.416 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T07:02:57.417 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T07:02:57.417 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi162.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T07:02:57.417 DEBUG:teuthology.orchestra.run.smithi162:> sudo apt-get clean 2024-09-16T07:02:57.493 DEBUG:teuthology.orchestra.run.smithi162:> sudo apt-get update 2024-09-16T07:02:57.626 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T07:02:57.626 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-16T07:02:57.631 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-16T07:02:57.649 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-16T07:02:59.512 INFO:teuthology.orchestra.run.smithi162.stdout:Reading package lists... 2024-09-16T07:02:59.540 DEBUG:teuthology.orchestra.run.smithi162:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T07:02:59.611 INFO:teuthology.orchestra.run.smithi162.stdout:Reading package lists... 2024-09-16T07:02:59.828 INFO:teuthology.orchestra.run.smithi162.stdout:Building dependency tree... 2024-09-16T07:02:59.828 INFO:teuthology.orchestra.run.smithi162.stdout:Reading state information... 2024-09-16T07:03:00.063 INFO:teuthology.orchestra.run.smithi162.stdout:linux-image-generic is already the newest version (5.4.0.195.193). 2024-09-16T07:03:00.064 INFO:teuthology.orchestra.run.smithi162.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-16T07:03:00.065 DEBUG:teuthology.orchestra.run.smithi162:> dpkg -s linux-image-generic 2024-09-16T07:03:00.088 INFO:teuthology.orchestra.run.smithi162.stdout:Package: linux-image-generic 2024-09-16T07:03:00.088 INFO:teuthology.orchestra.run.smithi162.stdout:Status: install ok installed 2024-09-16T07:03:00.088 INFO:teuthology.orchestra.run.smithi162.stdout:Priority: optional 2024-09-16T07:03:00.088 INFO:teuthology.orchestra.run.smithi162.stdout:Section: kernel 2024-09-16T07:03:00.088 INFO:teuthology.orchestra.run.smithi162.stdout:Installed-Size: 20 2024-09-16T07:03:00.088 INFO:teuthology.orchestra.run.smithi162.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T07:03:00.089 INFO:teuthology.orchestra.run.smithi162.stdout:Architecture: amd64 2024-09-16T07:03:00.089 INFO:teuthology.orchestra.run.smithi162.stdout:Source: linux-meta 2024-09-16T07:03:00.089 INFO:teuthology.orchestra.run.smithi162.stdout:Version: 5.4.0.195.193 2024-09-16T07:03:00.089 INFO:teuthology.orchestra.run.smithi162.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-16T07:03:00.089 INFO:teuthology.orchestra.run.smithi162.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T07:03:00.089 INFO:teuthology.orchestra.run.smithi162.stdout:Recommends: thermald 2024-09-16T07:03:00.089 INFO:teuthology.orchestra.run.smithi162.stdout:Description: Generic Linux kernel image 2024-09-16T07:03:00.089 INFO:teuthology.orchestra.run.smithi162.stdout: This package will always depend on the latest generic kernel image 2024-09-16T07:03:00.089 INFO:teuthology.orchestra.run.smithi162.stdout: available. 2024-09-16T07:03:00.090 DEBUG:teuthology.orchestra.run.smithi162:> mktemp 2024-09-16T07:03:00.094 INFO:teuthology.orchestra.run.smithi162.stdout:/tmp/tmp.oUo11ZA6l8 2024-09-16T07:03:00.095 DEBUG:teuthology.orchestra.run.smithi162:> sudo cp /boot/grub/grub.cfg /tmp/tmp.oUo11ZA6l8 2024-09-16T07:03:00.150 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0666 /tmp/tmp.oUo11ZA6l8 2024-09-16T07:03:00.185 DEBUG:teuthology.orchestra.remote:smithi162:/tmp/tmp.oUo11ZA6l8 is 9KB 2024-09-16T07:03:00.198 DEBUG:teuthology.orchestra.run.smithi162:> rm -fr /tmp/tmp.oUo11ZA6l8 2024-09-16T07:03:00.204 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T07:03:00.258 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-16T07:03:00.258 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T07:03:00.258 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T07:03:00.290 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-195-generic 2024-09-16T07:03:00.290 DEBUG:teuthology.orchestra.run.smithi162:> sudo update-grub 2024-09-16T07:03:01.739 INFO:teuthology.orchestra.run.smithi162.stderr:Sourcing file `/etc/default/grub' 2024-09-16T07:03:01.743 INFO:teuthology.orchestra.run.smithi162.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T07:03:01.750 INFO:teuthology.orchestra.run.smithi162.stderr:Generating grub configuration file ... 2024-09-16T07:03:02.026 INFO:teuthology.orchestra.run.smithi162.stderr:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-16T07:03:02.038 INFO:teuthology.orchestra.run.smithi162.stderr:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-16T07:03:02.408 INFO:teuthology.orchestra.run.smithi162.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-16T07:03:02.420 INFO:teuthology.orchestra.run.smithi162.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-16T07:03:03.376 INFO:teuthology.orchestra.run.smithi162.stderr:done 2024-09-16T07:03:03.378 DEBUG:teuthology.orchestra.run.smithi162:> sudo shutdown -r now 2024-09-16T07:03:33.410 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T07:03:33.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-16T07:03:33.413 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:04:07.998 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-09-16T07:04:17.005 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-16T07:04:17.006 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:04:49.137 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-09-16T07:04:50.317 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-16T07:04:50.318 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-195-generic"... 2024-09-16T07:04:50.318 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-09-16T07:04:50.368 INFO:teuthology.orchestra.run.smithi162.stdout:5.4.0-195-generic 2024-09-16T07:04:50.368 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-195-generic vs 5.4.0-195-generic 2024-09-16T07:04:50.368 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T07:04:50.368 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T07:04:51.369 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T07:04:51.370 DEBUG:teuthology.orchestra.run.smithi162:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T07:04:51.501 INFO:teuthology.orchestra.run.smithi162.stdout:ttyS1 2024-09-16T07:04:51.524 DEBUG:teuthology.parallel:result is None 2024-09-16T07:04:51.524 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T07:04:51.533 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T07:04:51.533 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T07:04:51.541 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T07:04:51.580 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T07:04:51.681 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T07:04:51.682 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T07:04:51.897 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T07:04:51.908 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T07:04:51.908 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T07:04:52.012 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:04:52.023 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:04:52.025 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T07:04:52.033 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T07:04:52.033 DEBUG:teuthology.orchestra.run.smithi162:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T07:04:52.050 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T07:04:52.074 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T07:04:52.074 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T07:04:52.100 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T07:04:52.107 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T07:04:52.114 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-16T07:04:52.115 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T07:04:52.181 DEBUG:teuthology.orchestra.run.smithi162:> sudo service rsyslog restart 2024-09-16T07:04:52.284 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T07:04:52.292 INFO:teuthology.task.internal:Starting timer... 2024-09-16T07:04:52.292 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T07:04:52.327 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T07:04:52.395 DEBUG:teuthology.task.selinux:Excluding smithi162: OS 'ubuntu' does not support SELinux 2024-09-16T07:04:52.396 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T07:04:52.396 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-16T07:04:52.396 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T07:04:52.396 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T07:04:52.450 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T07:04:52.463 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T07:04:52.464 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi162.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T07:10:25.367 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi162.front.sepia.ceph.com')] 2024-09-16T07:10:25.368 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-16T07:10:25.370 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:10:25.456 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-09-16T07:10:25.582 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-16T07:10:25.582 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T07:10:25.597 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T07:10:25.597 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T07:10:25.598 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T07:10:25.633 INFO:teuthology.orchestra.run.smithi162.stdout:16 Sep 07:10:25 ntpd[13727]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-16T07:10:25.633 INFO:teuthology.orchestra.run.smithi162.stdout:16 Sep 07:10:25 ntpd[13727]: Command line: ntpd -gq 2024-09-16T07:10:25.634 INFO:teuthology.orchestra.run.smithi162.stdout:16 Sep 07:10:25 ntpd[13727]: proto: precision = 0.068 usec (-24) 2024-09-16T07:10:25.634 INFO:teuthology.orchestra.run.smithi162.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T07:10:25.634 INFO:teuthology.orchestra.run.smithi162.stdout:16 Sep 07:10:25 ntpd[13727]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T07:10:25.635 INFO:teuthology.orchestra.run.smithi162.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T07:10:25.636 INFO:teuthology.orchestra.run.smithi162.stdout:16 Sep 07:10:25 ntpd[13727]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T07:10:25.636 INFO:teuthology.orchestra.run.smithi162.stdout:16 Sep 07:10:25 ntpd[13727]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T07:10:25.636 INFO:teuthology.orchestra.run.smithi162.stdout:16 Sep 07:10:25 ntpd[13727]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T07:10:25.636 INFO:teuthology.orchestra.run.smithi162.stdout:16 Sep 07:10:25 ntpd[13727]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T07:10:25.636 INFO:teuthology.orchestra.run.smithi162.stdout:16 Sep 07:10:25 ntpd[13727]: Listen normally on 3 enp3s0f1 172.21.15.162:123 2024-09-16T07:10:25.636 INFO:teuthology.orchestra.run.smithi162.stdout:16 Sep 07:10:25 ntpd[13727]: Listen normally on 4 lo [::1]:123 2024-09-16T07:10:25.636 INFO:teuthology.orchestra.run.smithi162.stdout:16 Sep 07:10:25 ntpd[13727]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f41%5]:123 2024-09-16T07:10:25.636 INFO:teuthology.orchestra.run.smithi162.stdout:16 Sep 07:10:25 ntpd[13727]: Listening on routing socket on fd #22 for interface updates 2024-09-16T07:10:26.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:26 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:10:26.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:26 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:10:26.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:26 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:10:26.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:26 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:10:26.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:26 ntpd[13727]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:10:26.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:26 ntpd[13727]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:10:27.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:27 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:10:27.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:27 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:10:27.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:27 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:10:27.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:27 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:10:27.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:27 ntpd[13727]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:10:27.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:27 ntpd[13727]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:10:28.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:28 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:10:28.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:28 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:10:28.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:28 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:10:28.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:28 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:10:28.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:28 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:10:28.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:28 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:10:28.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:28 ntpd[13727]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:10:28.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:28 ntpd[13727]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:10:28.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:28 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:10:28.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:28 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:10:28.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:28 ntpd[13727]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:10:28.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:28 ntpd[13727]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:10:29.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:29 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:10:29.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:29 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:10:29.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:29 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:10:29.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:29 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:10:29.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:29 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:10:29.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:29 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:10:29.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:29 ntpd[13727]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:10:29.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:29 ntpd[13727]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:10:30.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:30 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:10:30.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:30 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:10:30.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:30 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:10:30.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:30 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:10:30.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:30 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:10:30.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:30 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:10:30.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:30 ntpd[13727]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:10:30.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:30 ntpd[13727]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:10:30.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:30 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:10:30.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:30 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:10:30.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:30 ntpd[13727]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:10:30.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:30 ntpd[13727]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:10:31.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:31 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:10:31.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:31 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:10:31.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:31 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:10:31.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:31 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:10:31.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:31 ntpd[13727]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:10:31.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:31 ntpd[13727]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:10:32.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:32 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:10:32.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:32 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:10:32.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:32 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:10:32.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:32 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:10:32.634 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:32 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:10:32.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:32 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:10:32.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:32 ntpd[13727]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:10:32.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:32 ntpd[13727]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:10:32.635 INFO:teuthology.orchestra.run.smithi162.stdout:16 Sep 07:10:32 ntpd[13727]: ntpd: time slew +0.000212 s 2024-09-16T07:10:32.635 INFO:teuthology.orchestra.run.smithi162.stdout:ntpd: time slew +0.000212s 2024-09-16T07:10:32.635 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:32 ntpd[13727]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T07:10:32.636 INFO:teuthology.orchestra.run.smithi162.stderr:16 Sep 07:10:32 ntpd[13727]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T07:10:32.703 INFO:teuthology.orchestra.run.smithi162.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T07:10:32.703 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================== 2024-09-16T07:10:32.703 INFO:teuthology.orchestra.run.smithi162.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T07:10:32.703 INFO:teuthology.orchestra.run.smithi162.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T07:10:32.703 INFO:teuthology.orchestra.run.smithi162.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T07:10:32.703 INFO:teuthology.orchestra.run.smithi162.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T07:10:32.704 INFO:teuthology.run_tasks:Running task install... 2024-09-16T07:10:32.712 DEBUG:teuthology.task.install:project ceph 2024-09-16T07:10:32.712 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'}} 2024-09-16T07:10:32.712 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'} 2024-09-16T07:10:32.712 INFO:teuthology.task.install:Using flavor: default 2024-09-16T07:10:32.717 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T07:10:32.718 INFO:teuthology.task.install:extra packages: [] 2024-09-16T07:10:32.718 DEBUG:teuthology.orchestra.run.smithi162:> sudo apt-key list | grep Ceph 2024-09-16T07:10:32.822 INFO:teuthology.orchestra.run.smithi162.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-16T07:10:32.883 INFO:teuthology.orchestra.run.smithi162.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T07:10:32.883 INFO:teuthology.orchestra.run.smithi162.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T07:10:32.884 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T07:10:32.885 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T07:10:33.027 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default/ 2024-09-16T07:10:33.027 INFO:teuthology.task.install.deb:Package version is 18.2.4-845-ge4e227b3-1focal 2024-09-16T07:10:33.138 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-16T07:10:33.138 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T07:10:33.156 DEBUG:teuthology.orchestra.run.smithi162:> sudo apt-get update 2024-09-16T07:10:33.297 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T07:10:33.303 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-16T07:10:33.320 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-16T07:10:33.354 INFO:teuthology.orchestra.run.smithi162.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal InRelease 2024-09-16T07:10:33.375 INFO:teuthology.orchestra.run.smithi162.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-16T07:10:33.398 INFO:teuthology.orchestra.run.smithi162.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal Release.gpg 2024-09-16T07:10:33.416 INFO:teuthology.orchestra.run.smithi162.stdout:Hit:7 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-16T07:10:33.419 INFO:teuthology.orchestra.run.smithi162.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 Packages [17.6 kB] 2024-09-16T07:10:33.454 INFO:teuthology.orchestra.run.smithi162.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main i386 Packages [3,420 B] 2024-09-16T07:10:33.941 INFO:teuthology.orchestra.run.smithi162.stdout:Fetched 28.7 kB in 1s (45.9 kB/s) 2024-09-16T07:10:35.093 INFO:teuthology.orchestra.run.smithi162.stdout:Reading package lists... 2024-09-16T07:10:35.121 DEBUG:teuthology.orchestra.run.smithi162:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.4-845-ge4e227b3-1focal cephadm=18.2.4-845-ge4e227b3-1focal ceph-mds=18.2.4-845-ge4e227b3-1focal ceph-mgr=18.2.4-845-ge4e227b3-1focal ceph-common=18.2.4-845-ge4e227b3-1focal ceph-fuse=18.2.4-845-ge4e227b3-1focal ceph-test=18.2.4-845-ge4e227b3-1focal ceph-volume=18.2.4-845-ge4e227b3-1focal radosgw=18.2.4-845-ge4e227b3-1focal python3-rados=18.2.4-845-ge4e227b3-1focal python3-rgw=18.2.4-845-ge4e227b3-1focal python3-cephfs=18.2.4-845-ge4e227b3-1focal python3-rbd=18.2.4-845-ge4e227b3-1focal libcephfs2=18.2.4-845-ge4e227b3-1focal libcephfs-dev=18.2.4-845-ge4e227b3-1focal librados2=18.2.4-845-ge4e227b3-1focal librbd1=18.2.4-845-ge4e227b3-1focal rbd-fuse=18.2.4-845-ge4e227b3-1focal 2024-09-16T07:10:35.193 INFO:teuthology.orchestra.run.smithi162.stdout:Reading package lists... 2024-09-16T07:10:35.410 INFO:teuthology.orchestra.run.smithi162.stdout:Building dependency tree... 2024-09-16T07:10:35.410 INFO:teuthology.orchestra.run.smithi162.stdout:Reading state information... 2024-09-16T07:10:35.613 INFO:teuthology.orchestra.run.smithi162.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T07:10:35.615 INFO:teuthology.orchestra.run.smithi162.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-16T07:10:35.615 INFO:teuthology.orchestra.run.smithi162.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T07:10:35.615 INFO:teuthology.orchestra.run.smithi162.stdout:The following additional packages will be installed: 2024-09-16T07:10:35.615 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T07:10:35.616 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-16T07:10:35.616 INFO:teuthology.orchestra.run.smithi162.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-16T07:10:35.616 INFO:teuthology.orchestra.run.smithi162.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-16T07:10:35.616 INFO:teuthology.orchestra.run.smithi162.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-16T07:10:35.618 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-16T07:10:35.618 INFO:teuthology.orchestra.run.smithi162.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2024-09-16T07:10:35.618 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2024-09-16T07:10:35.618 INFO:teuthology.orchestra.run.smithi162.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-16T07:10:35.618 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2024-09-16T07:10:35.618 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2024-09-16T07:10:35.618 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-16T07:10:35.618 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-16T07:10:35.618 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-16T07:10:35.618 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-16T07:10:35.618 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-16T07:10:35.619 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T07:10:35.619 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy python3-simplegeneric python3-simplejson 2024-09-16T07:10:35.619 INFO:teuthology.orchestra.run.smithi162.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-16T07:10:35.619 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-16T07:10:35.619 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-16T07:10:35.619 INFO:teuthology.orchestra.run.smithi162.stdout: qttranslations5-l10n socat xmlstarlet 2024-09-16T07:10:35.619 INFO:teuthology.orchestra.run.smithi162.stdout:Suggested packages: 2024-09-16T07:10:35.619 INFO:teuthology.orchestra.run.smithi162.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-16T07:10:35.619 INFO:teuthology.orchestra.run.smithi162.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-16T07:10:35.619 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-16T07:10:35.619 INFO:teuthology.orchestra.run.smithi162.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-16T07:10:35.619 INFO:teuthology.orchestra.run.smithi162.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-16T07:10:35.619 INFO:teuthology.orchestra.run.smithi162.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-16T07:10:35.619 INFO:teuthology.orchestra.run.smithi162.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-16T07:10:35.620 INFO:teuthology.orchestra.run.smithi162.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T07:10:35.620 INFO:teuthology.orchestra.run.smithi162.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-16T07:10:35.620 INFO:teuthology.orchestra.run.smithi162.stdout:Recommended packages: 2024-09-16T07:10:35.620 INFO:teuthology.orchestra.run.smithi162.stdout: btrfs-tools 2024-09-16T07:10:35.740 INFO:teuthology.orchestra.run.smithi162.stdout:The following NEW packages will be installed: 2024-09-16T07:10:35.741 INFO:teuthology.orchestra.run.smithi162.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T07:10:35.741 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T07:10:35.741 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-16T07:10:35.741 INFO:teuthology.orchestra.run.smithi162.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-16T07:10:35.741 INFO:teuthology.orchestra.run.smithi162.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-16T07:10:35.742 INFO:teuthology.orchestra.run.smithi162.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T07:10:35.742 INFO:teuthology.orchestra.run.smithi162.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-16T07:10:35.742 INFO:teuthology.orchestra.run.smithi162.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-asyncssh-doc 2024-09-16T07:10:35.742 INFO:teuthology.orchestra.run.smithi162.stdout: python-natsort-doc python-pastedeploy-tpl python3-asyncssh 2024-09-16T07:10:35.742 INFO:teuthology.orchestra.run.smithi162.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2024-09-16T07:10:35.743 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-16T07:10:35.743 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2024-09-16T07:10:35.743 INFO:teuthology.orchestra.run.smithi162.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2024-09-16T07:10:35.743 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-16T07:10:35.743 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-16T07:10:35.744 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-16T07:10:35.744 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-16T07:10:35.744 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-16T07:10:35.744 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2024-09-16T07:10:35.744 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2024-09-16T07:10:35.745 INFO:teuthology.orchestra.run.smithi162.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-09-16T07:10:35.745 INFO:teuthology.orchestra.run.smithi162.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2024-09-16T07:10:35.745 INFO:teuthology.orchestra.run.smithi162.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2024-09-16T07:10:35.745 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse socat 2024-09-16T07:10:35.745 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet 2024-09-16T07:10:35.746 INFO:teuthology.orchestra.run.smithi162.stdout:The following packages will be upgraded: 2024-09-16T07:10:35.746 INFO:teuthology.orchestra.run.smithi162.stdout: librados2 librbd1 2024-09-16T07:10:35.805 INFO:teuthology.orchestra.run.smithi162.stdout:2 upgraded, 110 newly installed, 0 to remove and 295 not upgraded. 2024-09-16T07:10:35.805 INFO:teuthology.orchestra.run.smithi162.stdout:Need to get 199 MB of archives. 2024-09-16T07:10:35.805 INFO:teuthology.orchestra.run.smithi162.stdout:After this operation, 977 MB of additional disk space will be used. 2024-09-16T07:10:35.805 INFO:teuthology.orchestra.run.smithi162.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-16T07:10:35.887 INFO:teuthology.orchestra.run.smithi162.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-16T07:10:35.889 INFO:teuthology.orchestra.run.smithi162.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 18.2.4-845-ge4e227b3-1focal [3,451 kB] 2024-09-16T07:10:35.890 INFO:teuthology.orchestra.run.smithi162.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-16T07:10:35.907 INFO:teuthology.orchestra.run.smithi162.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-16T07:10:35.960 INFO:teuthology.orchestra.run.smithi162.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-16T07:10:35.962 INFO:teuthology.orchestra.run.smithi162.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-16T07:10:35.968 INFO:teuthology.orchestra.run.smithi162.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-16T07:10:35.970 INFO:teuthology.orchestra.run.smithi162.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-16T07:10:35.971 INFO:teuthology.orchestra.run.smithi162.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-16T07:10:35.972 INFO:teuthology.orchestra.run.smithi162.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-16T07:10:35.972 INFO:teuthology.orchestra.run.smithi162.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-16T07:10:35.975 INFO:teuthology.orchestra.run.smithi162.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-16T07:10:35.975 INFO:teuthology.orchestra.run.smithi162.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-16T07:10:35.977 INFO:teuthology.orchestra.run.smithi162.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-16T07:10:35.994 INFO:teuthology.orchestra.run.smithi162.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-16T07:10:35.994 INFO:teuthology.orchestra.run.smithi162.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-16T07:10:35.995 INFO:teuthology.orchestra.run.smithi162.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-16T07:10:35.996 INFO:teuthology.orchestra.run.smithi162.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T07:10:35.996 INFO:teuthology.orchestra.run.smithi162.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-16T07:10:35.996 INFO:teuthology.orchestra.run.smithi162.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-16T07:10:35.998 INFO:teuthology.orchestra.run.smithi162.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-16T07:10:35.998 INFO:teuthology.orchestra.run.smithi162.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-16T07:10:35.999 INFO:teuthology.orchestra.run.smithi162.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-16T07:10:36.011 INFO:teuthology.orchestra.run.smithi162.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-16T07:10:36.018 INFO:teuthology.orchestra.run.smithi162.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-16T07:10:36.025 INFO:teuthology.orchestra.run.smithi162.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-16T07:10:36.025 INFO:teuthology.orchestra.run.smithi162.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-16T07:10:36.026 INFO:teuthology.orchestra.run.smithi162.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-16T07:10:36.026 INFO:teuthology.orchestra.run.smithi162.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-16T07:10:36.028 INFO:teuthology.orchestra.run.smithi162.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-16T07:10:36.031 INFO:teuthology.orchestra.run.smithi162.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-16T07:10:36.031 INFO:teuthology.orchestra.run.smithi162.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-16T07:10:36.035 INFO:teuthology.orchestra.run.smithi162.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-16T07:10:36.044 INFO:teuthology.orchestra.run.smithi162.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2024-09-16T07:10:36.045 INFO:teuthology.orchestra.run.smithi162.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-16T07:10:36.046 INFO:teuthology.orchestra.run.smithi162.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-16T07:10:36.047 INFO:teuthology.orchestra.run.smithi162.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-16T07:10:36.054 INFO:teuthology.orchestra.run.smithi162.stdout:Get:39 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 18.2.4-845-ge4e227b3-1focal [4,042 kB] 2024-09-16T07:10:36.061 INFO:teuthology.orchestra.run.smithi162.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-16T07:10:36.061 INFO:teuthology.orchestra.run.smithi162.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-16T07:10:36.063 INFO:teuthology.orchestra.run.smithi162.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-16T07:10:36.063 INFO:teuthology.orchestra.run.smithi162.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-16T07:10:36.064 INFO:teuthology.orchestra.run.smithi162.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-16T07:10:36.065 INFO:teuthology.orchestra.run.smithi162.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-16T07:10:36.078 INFO:teuthology.orchestra.run.smithi162.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T07:10:36.078 INFO:teuthology.orchestra.run.smithi162.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-16T07:10:36.079 INFO:teuthology.orchestra.run.smithi162.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-16T07:10:36.081 INFO:teuthology.orchestra.run.smithi162.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-16T07:10:36.094 INFO:teuthology.orchestra.run.smithi162.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-16T07:10:36.095 INFO:teuthology.orchestra.run.smithi162.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-16T07:10:36.095 INFO:teuthology.orchestra.run.smithi162.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-16T07:10:36.112 INFO:teuthology.orchestra.run.smithi162.stdout:Get:53 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 18.2.4-845-ge4e227b3-1focal [746 kB] 2024-09-16T07:10:36.123 INFO:teuthology.orchestra.run.smithi162.stdout:Get:54 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 18.2.4-845-ge4e227b3-1focal [387 kB] 2024-09-16T07:10:36.130 INFO:teuthology.orchestra.run.smithi162.stdout:Get:55 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 18.2.4-845-ge4e227b3-1focal [31.9 kB] 2024-09-16T07:10:36.130 INFO:teuthology.orchestra.run.smithi162.stdout:Get:56 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 18.2.4-845-ge4e227b3-1focal [200 kB] 2024-09-16T07:10:36.133 INFO:teuthology.orchestra.run.smithi162.stdout:Get:57 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 18.2.4-845-ge4e227b3-1focal [62.5 kB] 2024-09-16T07:10:36.134 INFO:teuthology.orchestra.run.smithi162.stdout:Get:58 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 18.2.4-845-ge4e227b3-1focal [368 kB] 2024-09-16T07:10:36.140 INFO:teuthology.orchestra.run.smithi162.stdout:Get:59 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 18.2.4-845-ge4e227b3-1focal [6,630 kB] 2024-09-16T07:10:36.179 INFO:teuthology.orchestra.run.smithi162.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-16T07:10:36.265 INFO:teuthology.orchestra.run.smithi162.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-16T07:10:36.265 INFO:teuthology.orchestra.run.smithi162.stdout:Get:62 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 18.2.4-845-ge4e227b3-1focal [113 kB] 2024-09-16T07:10:36.266 INFO:teuthology.orchestra.run.smithi162.stdout:Get:63 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 18.2.4-845-ge4e227b3-1focal [478 kB] 2024-09-16T07:10:36.321 INFO:teuthology.orchestra.run.smithi162.stdout:Get:64 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 18.2.4-845-ge4e227b3-1focal [25.2 MB] 2024-09-16T07:10:36.321 INFO:teuthology.orchestra.run.smithi162.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-16T07:10:36.338 INFO:teuthology.orchestra.run.smithi162.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-16T07:10:36.338 INFO:teuthology.orchestra.run.smithi162.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-16T07:10:36.338 INFO:teuthology.orchestra.run.smithi162.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-16T07:10:36.339 INFO:teuthology.orchestra.run.smithi162.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-16T07:10:36.339 INFO:teuthology.orchestra.run.smithi162.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-16T07:10:36.339 INFO:teuthology.orchestra.run.smithi162.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-16T07:10:36.340 INFO:teuthology.orchestra.run.smithi162.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-16T07:10:36.341 INFO:teuthology.orchestra.run.smithi162.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-16T07:10:36.341 INFO:teuthology.orchestra.run.smithi162.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-16T07:10:36.356 INFO:teuthology.orchestra.run.smithi162.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-16T07:10:36.359 INFO:teuthology.orchestra.run.smithi162.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-16T07:10:36.373 INFO:teuthology.orchestra.run.smithi162.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-16T07:10:36.375 INFO:teuthology.orchestra.run.smithi162.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-16T07:10:36.376 INFO:teuthology.orchestra.run.smithi162.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-16T07:10:36.380 INFO:teuthology.orchestra.run.smithi162.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-16T07:10:36.384 INFO:teuthology.orchestra.run.smithi162.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-16T07:10:36.401 INFO:teuthology.orchestra.run.smithi162.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-16T07:10:36.404 INFO:teuthology.orchestra.run.smithi162.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2024-09-16T07:10:36.406 INFO:teuthology.orchestra.run.smithi162.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-16T07:10:36.442 INFO:teuthology.orchestra.run.smithi162.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-16T07:10:36.443 INFO:teuthology.orchestra.run.smithi162.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-16T07:10:36.443 INFO:teuthology.orchestra.run.smithi162.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-16T07:10:36.444 INFO:teuthology.orchestra.run.smithi162.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-16T07:10:36.456 INFO:teuthology.orchestra.run.smithi162.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T07:10:36.456 INFO:teuthology.orchestra.run.smithi162.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-16T07:10:36.457 INFO:teuthology.orchestra.run.smithi162.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-16T07:10:36.457 INFO:teuthology.orchestra.run.smithi162.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-16T07:10:36.458 INFO:teuthology.orchestra.run.smithi162.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-16T07:10:36.776 INFO:teuthology.orchestra.run.smithi162.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 18.2.4-845-ge4e227b3-1focal [9,611 kB] 2024-09-16T07:10:36.888 INFO:teuthology.orchestra.run.smithi162.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 18.2.4-845-ge4e227b3-1focal [238 kB] 2024-09-16T07:10:36.891 INFO:teuthology.orchestra.run.smithi162.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 18.2.4-845-ge4e227b3-1focal [148 kB] 2024-09-16T07:10:36.892 INFO:teuthology.orchestra.run.smithi162.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 18.2.4-845-ge4e227b3-1focal [1,574 kB] 2024-09-16T07:10:36.919 INFO:teuthology.orchestra.run.smithi162.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 18.2.4-845-ge4e227b3-1focal [6,640 kB] 2024-09-16T07:10:37.029 INFO:teuthology.orchestra.run.smithi162.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 18.2.4-845-ge4e227b3-1focal [24.5 MB] 2024-09-16T07:10:37.520 INFO:teuthology.orchestra.run.smithi162.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 18.2.4-845-ge4e227b3-1focal [13.7 kB] 2024-09-16T07:10:37.535 INFO:teuthology.orchestra.run.smithi162.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 18.2.4-845-ge4e227b3-1focal [856 kB] 2024-09-16T07:10:37.546 INFO:teuthology.orchestra.run.smithi162.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 18.2.4-845-ge4e227b3-1focal [2,295 kB] 2024-09-16T07:10:37.563 INFO:teuthology.orchestra.run.smithi162.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 18.2.4-845-ge4e227b3-1focal [239 kB] 2024-09-16T07:10:37.565 INFO:teuthology.orchestra.run.smithi162.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 18.2.4-845-ge4e227b3-1focal [143 kB] 2024-09-16T07:10:37.568 INFO:teuthology.orchestra.run.smithi162.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 18.2.4-845-ge4e227b3-1focal [7,015 kB] 2024-09-16T07:10:37.661 INFO:teuthology.orchestra.run.smithi162.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 18.2.4-845-ge4e227b3-1focal [7,882 kB] 2024-09-16T07:10:37.791 INFO:teuthology.orchestra.run.smithi162.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 18.2.4-845-ge4e227b3-1focal [13.9 kB] 2024-09-16T07:10:37.791 INFO:teuthology.orchestra.run.smithi162.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 18.2.4-845-ge4e227b3-1focal [46.9 MB] 2024-09-16T07:10:38.827 INFO:teuthology.orchestra.run.smithi162.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 18.2.4-845-ge4e227b3-1focal [120 kB] 2024-09-16T07:10:38.829 INFO:teuthology.orchestra.run.smithi162.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 18.2.4-845-ge4e227b3-1focal [37.4 kB] 2024-09-16T07:10:38.831 INFO:teuthology.orchestra.run.smithi162.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 18.2.4-845-ge4e227b3-1focal [13.5 MB] 2024-09-16T07:10:39.076 INFO:teuthology.orchestra.run.smithi162.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 18.2.4-845-ge4e227b3-1focal [107 kB] 2024-09-16T07:10:39.730 INFO:teuthology.orchestra.run.smithi162.stdout:Fetched 199 MB in 3s (59.8 MB/s) 2024-09-16T07:10:39.901 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-16T07:10:39.958 INFO:teuthology.orchestra.run.smithi162.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-16T07:10:39.962 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-16T07:10:40.092 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-16T07:10:40.437 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T07:10:40.456 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-16T07:10:40.495 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-16T07:10:40.840 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T07:10:40.857 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-16T07:10:40.890 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-16T07:10:41.351 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T07:10:41.367 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T07:10:41.420 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T07:10:41.955 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T07:10:41.973 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T07:10:42.013 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T07:10:42.308 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T07:10:42.323 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T07:10:42.357 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T07:10:42.752 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-16T07:10:42.769 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-16T07:10:42.810 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-16T07:10:43.213 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../007-librbd1_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:10:43.339 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking librbd1 (18.2.4-845-ge4e227b3-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-16T07:10:44.025 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../008-librados2_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:10:44.126 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking librados2 (18.2.4-845-ge4e227b3-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-16T07:10:44.839 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T07:10:44.856 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../009-libcephfs2_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:10:44.930 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libcephfs2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:10:45.266 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-rados. 2024-09-16T07:10:45.282 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../010-python3-rados_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:10:45.324 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-rados (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:10:45.677 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T07:10:45.693 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../011-python3-ceph-argparse_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T07:10:45.727 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-ceph-argparse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:10:45.980 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T07:10:45.997 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../012-python3-cephfs_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:10:46.054 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-cephfs (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:10:46.399 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T07:10:46.415 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../013-python3-ceph-common_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T07:10:46.457 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:10:46.810 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T07:10:46.826 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2024-09-16T07:10:46.868 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-16T07:10:47.130 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T07:10:47.146 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../015-python3-rbd_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:10:47.188 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-rbd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:10:47.557 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-16T07:10:47.576 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-16T07:10:47.615 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-16T07:10:48.010 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-16T07:10:48.026 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-16T07:10:48.068 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-16T07:10:48.413 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T07:10:48.429 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-16T07:10:48.471 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-16T07:10:48.824 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package librgw2. 2024-09-16T07:10:48.843 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../019-librgw2_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:10:48.882 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking librgw2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:10:49.793 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T07:10:49.809 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../020-python3-rgw_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:10:49.851 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-rgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:10:50.213 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package liboath0. 2024-09-16T07:10:50.229 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-16T07:10:50.271 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-16T07:10:50.590 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T07:10:50.608 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../022-libradosstriper1_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:10:50.640 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libradosstriper1 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:10:51.119 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-common. 2024-09-16T07:10:51.134 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../023-ceph-common_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:10:51.177 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:10:53.657 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-base. 2024-09-16T07:10:53.673 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../024-ceph-base_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:10:53.725 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-base (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:10:55.137 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T07:10:55.156 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-16T07:10:55.195 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-16T07:10:55.548 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T07:10:55.568 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../026-python3-natsort_7.0.1-1_all.deb ... 2024-09-16T07:10:55.606 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-16T07:10:55.943 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T07:10:55.963 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2024-09-16T07:10:56.001 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-16T07:10:56.304 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T07:10:56.321 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-16T07:10:56.362 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-16T07:10:56.673 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-mako. 2024-09-16T07:10:56.692 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-16T07:10:56.732 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-16T07:10:57.051 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T07:10:57.068 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T07:10:57.135 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T07:10:57.446 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T07:10:57.465 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-16T07:10:57.504 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-16T07:10:57.815 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-webob. 2024-09-16T07:10:57.833 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-16T07:10:57.873 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-16T07:10:58.176 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-16T07:10:58.196 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-16T07:10:58.234 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-16T07:10:58.563 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-bs4. 2024-09-16T07:10:58.578 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2024-09-16T07:10:58.621 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-16T07:10:58.991 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T07:10:59.006 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-16T07:10:59.052 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-16T07:10:59.360 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T07:10:59.380 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2024-09-16T07:10:59.418 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-16T07:10:59.748 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-paste. 2024-09-16T07:10:59.762 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-16T07:10:59.804 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-16T07:11:00.116 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T07:11:00.135 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-16T07:11:00.208 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-16T07:11:00.510 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T07:11:00.526 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-16T07:11:00.568 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-16T07:11:00.882 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T07:11:00.900 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2024-09-16T07:11:00.940 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-16T07:11:01.260 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T07:11:01.275 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-16T07:11:01.317 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-16T07:11:01.690 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T07:11:01.706 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-16T07:11:01.748 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-16T07:11:02.060 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T07:11:02.077 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../043-ceph-mgr-modules-core_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T07:11:02.118 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-mgr-modules-core (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:11:02.448 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T07:11:02.468 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../044-libsqlite3-mod-ceph_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:11:02.506 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libsqlite3-mod-ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:11:02.859 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T07:11:02.875 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../045-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-16T07:11:02.917 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-16T07:11:03.229 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-cryptography. 2024-09-16T07:11:03.248 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../046-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-16T07:11:03.287 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-16T07:11:03.667 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T07:11:03.685 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../047-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-16T07:11:03.725 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-16T07:11:04.116 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T07:11:04.131 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../048-ceph-mgr_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:11:04.173 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-mgr (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:11:04.660 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T07:11:04.677 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../049-ceph-mon_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:11:04.710 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-mon (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:11:05.738 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T07:11:05.755 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../050-ceph-osd_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:11:05.797 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-osd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:11:07.900 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph. 2024-09-16T07:11:07.916 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../051-ceph_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:11:07.958 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:11:08.277 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T07:11:08.296 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../052-ceph-fuse_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:11:08.335 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:11:08.739 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T07:11:08.757 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../053-ceph-mds_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:11:08.798 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-mds (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:11:09.343 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package cephadm. 2024-09-16T07:11:09.359 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../054-cephadm_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:11:09.401 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:11:09.937 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T07:11:09.956 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../055-python3-asyncssh_1.12.2-1_all.deb ... 2024-09-16T07:11:10.560 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2024-09-16T07:11:10.893 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T07:11:10.906 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-16T07:11:10.940 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-16T07:11:11.252 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T07:11:11.272 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T07:11:11.302 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-mgr-cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:11:11.639 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-jwt. 2024-09-16T07:11:11.654 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../058-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-16T07:11:11.705 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-16T07:11:12.050 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-attr. 2024-09-16T07:11:12.066 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../059-python3-attr_19.3.0-2_all.deb ... 2024-09-16T07:11:12.108 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-16T07:11:12.470 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T07:11:12.486 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../060-python3-pluggy_0.13.0-2_all.deb ... 2024-09-16T07:11:12.529 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-16T07:11:12.839 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-py. 2024-09-16T07:11:12.859 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../061-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-16T07:11:12.897 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-16T07:11:13.242 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-16T07:11:13.258 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../062-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-16T07:11:13.300 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-16T07:11:13.603 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-packaging. 2024-09-16T07:11:13.621 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../063-python3-packaging_20.3-1_all.deb ... 2024-09-16T07:11:13.678 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-16T07:11:13.973 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T07:11:13.992 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../064-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-16T07:11:14.031 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-16T07:11:14.342 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T07:11:14.359 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../065-python3-pytest_4.6.9-1_all.deb ... 2024-09-16T07:11:14.392 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-16T07:11:14.713 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T07:11:14.732 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../066-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T07:11:14.771 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T07:11:15.050 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libjs-underscore. 2024-09-16T07:11:15.066 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../067-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-16T07:11:15.107 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-16T07:11:15.385 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-16T07:11:15.402 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../068-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-16T07:11:15.468 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-16T07:11:15.788 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-routes. 2024-09-16T07:11:15.804 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../069-python3-routes_2.4.1-2_all.deb ... 2024-09-16T07:11:15.846 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-16T07:11:16.199 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T07:11:16.216 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T07:11:16.257 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-mgr-dashboard (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:11:17.663 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-decorator. 2024-09-16T07:11:17.681 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../071-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-16T07:11:17.721 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-16T07:11:18.066 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-16T07:11:18.082 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../072-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-16T07:11:18.124 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-16T07:11:18.427 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-scipy. 2024-09-16T07:11:18.443 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../073-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-16T07:11:18.493 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-16T07:11:19.811 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-16T07:11:19.827 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-16T07:11:19.869 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-16T07:11:20.398 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T07:11:20.414 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../075-python3-joblib_0.14.0-3_all.deb ... 2024-09-16T07:11:20.448 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-16T07:11:20.801 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T07:11:20.818 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../076-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-16T07:11:20.860 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-16T07:11:21.611 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T07:11:21.629 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../077-ceph-mgr-diskprediction-local_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T07:11:21.669 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:11:22.805 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T07:11:22.825 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../078-python3-cachetools_4.0.0-1_all.deb ... 2024-09-16T07:11:22.864 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-16T07:11:23.167 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T07:11:23.187 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../079-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-16T07:11:23.225 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-16T07:11:23.562 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T07:11:23.579 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../080-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-16T07:11:23.620 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-16T07:11:24.044 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T07:11:24.059 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../081-python3-rsa_4.0-3_all.deb ... 2024-09-16T07:11:24.103 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-16T07:11:24.401 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T07:11:24.418 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../082-python3-google-auth_1.5.1-2_all.deb ... 2024-09-16T07:11:24.459 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-16T07:11:24.796 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-blinker. 2024-09-16T07:11:24.816 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../083-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-16T07:11:24.854 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-16T07:11:25.158 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-16T07:11:25.177 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../084-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-16T07:11:25.258 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-16T07:11:25.578 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T07:11:25.596 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../085-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-16T07:11:25.628 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-16T07:11:26.257 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T07:11:26.274 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../086-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-16T07:11:26.315 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-16T07:11:26.660 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T07:11:26.678 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../087-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-16T07:11:26.718 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-16T07:11:27.262 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T07:11:27.281 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../088-ceph-mgr-k8sevents_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T07:11:27.320 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-mgr-k8sevents (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:11:27.675 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T07:11:27.691 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../089-libonig5_6.9.4-1_amd64.deb ... 2024-09-16T07:11:27.725 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-16T07:11:28.073 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T07:11:28.087 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../090-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-16T07:11:28.129 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-16T07:11:28.449 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package jq. 2024-09-16T07:11:28.466 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../091-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-16T07:11:28.507 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-16T07:11:28.743 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package socat. 2024-09-16T07:11:28.761 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../092-socat_1.7.3.3-2_amd64.deb ... 2024-09-16T07:11:28.801 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-16T07:11:29.130 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T07:11:29.146 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-16T07:11:29.187 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-16T07:11:29.534 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-test. 2024-09-16T07:11:29.551 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../094-ceph-test_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:11:29.592 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-test (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:11:33.788 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T07:11:33.804 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../095-ceph-volume_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T07:11:33.846 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking ceph-volume (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:11:34.143 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-16T07:11:34.159 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../096-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-16T07:11:34.202 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-16T07:11:34.705 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T07:11:34.722 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../097-libcephfs-dev_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:11:34.763 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libcephfs-dev (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:11:35.167 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T07:11:35.188 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../098-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-16T07:11:35.243 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-16T07:11:35.584 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T07:11:35.600 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../099-python-asyncssh-doc_1.12.2-1_all.deb ... 2024-09-16T07:11:35.642 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2024-09-16T07:11:36.011 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-16T07:11:36.027 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../100-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-16T07:11:36.069 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-16T07:11:37.724 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-16T07:11:37.743 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../101-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-16T07:11:37.782 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-16T07:11:38.119 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-webencodings. 2024-09-16T07:11:38.135 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-16T07:11:38.177 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-16T07:11:38.480 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-html5lib. 2024-09-16T07:11:38.498 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2024-09-16T07:11:38.537 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-16T07:11:38.899 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-16T07:11:38.918 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-16T07:11:38.957 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-16T07:11:39.428 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T07:11:39.446 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../105-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T07:11:39.494 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T07:11:39.873 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T07:11:39.918 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../106-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-16T07:11:39.931 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-16T07:11:40.251 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T07:11:40.268 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-16T07:11:40.309 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-16T07:11:40.640 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T07:11:40.661 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../108-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-16T07:11:40.698 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-16T07:11:40.984 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T07:11:41.002 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../109-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-16T07:11:41.067 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-16T07:11:41.747 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package radosgw. 2024-09-16T07:11:41.764 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../110-radosgw_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:11:41.805 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking radosgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:11:43.208 INFO:teuthology.orchestra.run.smithi162.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T07:11:43.224 INFO:teuthology.orchestra.run.smithi162.stdout:Preparing to unpack .../111-rbd-fuse_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:11:43.265 INFO:teuthology.orchestra.run.smithi162.stdout:Unpacking rbd-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:11:43.586 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-16T07:11:43.890 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-16T07:11:44.016 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-16T07:11:44.159 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-16T07:11:44.369 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:11:44.954 INFO:teuthology.orchestra.run.smithi162.stdout:Adding system user cephadm....done 2024-09-16T07:11:45.109 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-16T07:11:45.236 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-16T07:11:45.621 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T07:11:45.931 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-16T07:11:46.057 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-16T07:11:46.183 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-16T07:11:46.518 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-16T07:11:46.912 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-16T07:11:47.222 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-16T07:11:47.515 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-16T07:11:47.799 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-16T07:11:48.076 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-ceph-argparse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:11:48.368 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-16T07:11:48.494 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-16T07:11:48.787 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-16T07:11:49.063 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-16T07:11:49.189 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-16T07:11:49.574 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-16T07:11:49.859 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-16T07:11:50.177 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-16T07:11:50.528 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-16T07:11:50.796 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-16T07:11:51.122 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-16T07:11:51.423 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-16T07:11:51.708 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-16T07:11:51.834 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-16T07:11:51.994 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-16T07:11:52.286 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-16T07:11:52.680 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-16T07:11:53.023 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-16T07:11:53.383 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-16T07:11:53.501 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:11:53.836 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T07:11:53.970 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-16T07:11:54.297 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-16T07:11:54.448 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-16T07:11:54.574 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-16T07:11:54.859 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-16T07:11:55.160 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T07:11:55.270 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-16T07:11:55.396 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-16T07:11:55.731 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-16T07:11:56.015 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-16T07:11:56.316 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-16T07:11:56.443 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T07:11:56.719 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-16T07:11:57.245 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-16T07:11:57.554 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-16T07:11:57.956 INFO:teuthology.orchestra.run.smithi162.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-16T07:11:58.190 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-16T07:11:58.317 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-16T07:11:58.651 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-16T07:11:59.003 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-16T07:11:59.129 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-16T07:11:59.255 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-16T07:11:59.615 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-16T07:11:59.941 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-16T07:12:01.414 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-16T07:12:01.540 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-16T07:12:01.916 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-16T07:12:02.220 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-16T07:12:02.502 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T07:12:02.628 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-16T07:12:02.980 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-16T07:12:03.307 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2024-09-16T07:12:03.675 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-16T07:12:04.009 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-16T07:12:05.048 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-16T07:12:05.174 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-16T07:12:05.301 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-16T07:12:05.460 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-16T07:12:05.745 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-16T07:12:05.871 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-16T07:12:06.164 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-16T07:12:06.448 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T07:12:06.767 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-16T07:12:07.136 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up librados2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:07.278 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2024-09-16T07:12:07.404 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up librgw2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:07.530 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libsqlite3-mod-ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:07.734 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libcephfs2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:07.876 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-16T07:12:08.170 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libradosstriper1 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:08.296 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-16T07:12:08.589 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up librbd1 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:08.716 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-mgr-modules-core (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:08.844 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:09.040 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T07:12:09.040 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T07:12:09.487 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up libcephfs-dev (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:09.588 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-rados (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:09.722 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-16T07:12:10.618 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-rbd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:10.752 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up rbd-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:10.911 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-rgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:11.012 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up python3-cephfs (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:11.113 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:11.666 INFO:teuthology.orchestra.run.smithi162.stdout:Adding group ceph....done 2024-09-16T07:12:12.026 INFO:teuthology.orchestra.run.smithi162.stdout:Adding system user ceph....done 2024-09-16T07:12:12.219 INFO:teuthology.orchestra.run.smithi162.stdout:Setting system user ceph properties....done 2024-09-16T07:12:12.245 INFO:teuthology.orchestra.run.smithi162.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T07:12:12.394 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T07:12:12.715 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T07:12:13.157 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-test (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:13.316 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up radosgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:13.818 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T07:12:13.819 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T07:12:14.268 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-base (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:14.656 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T07:12:15.120 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-mds (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:15.362 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T07:12:15.362 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T07:12:15.813 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-mgr (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:16.050 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T07:12:16.050 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T07:12:16.489 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-osd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:16.980 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T07:12:16.981 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T07:12:17.424 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-mgr-k8sevents (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:17.693 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-mgr-diskprediction-local (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:17.928 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-mon (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:18.152 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T07:12:18.153 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T07:12:18.629 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-mgr-cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:18.797 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:18.922 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-mgr-dashboard (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:19.065 INFO:teuthology.orchestra.run.smithi162.stdout:Setting up ceph-volume (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:12:19.410 INFO:teuthology.orchestra.run.smithi162.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-16T07:12:19.562 INFO:teuthology.orchestra.run.smithi162.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-16T07:12:19.838 INFO:teuthology.orchestra.run.smithi162.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-16T07:12:22.649 INFO:teuthology.orchestra.run.smithi162.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-16T07:12:23.461 INFO:teuthology.orchestra.run.smithi162.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T07:12:23.466 DEBUG:teuthology.parallel:result is None 2024-09-16T07:12:23.467 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T07:12:23.611 DEBUG:teuthology.orchestra.run.smithi162:> dpkg-query -W -f '${Version}' ceph 2024-09-16T07:12:23.635 INFO:teuthology.orchestra.run.smithi162.stdout:18.2.4-845-ge4e227b3-1focal 2024-09-16T07:12:23.636 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-845-ge4e227b3-1focal 2024-09-16T07:12:23.636 INFO:teuthology.task.install:The correct ceph version 18.2.4-845-ge4e227b3-1focal is installed. 2024-09-16T07:12:23.638 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T07:12:23.638 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-16T07:12:23.638 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T07:12:23.650 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T07:12:23.650 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-16T07:12:23.650 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T07:12:23.704 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T07:12:23.715 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T07:12:23.715 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-16T07:12:23.715 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T07:12:23.732 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T07:12:23.745 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T07:12:23.745 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-16T07:12:23.745 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T07:12:23.802 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T07:12:23.811 INFO:teuthology.run_tasks:Running task workunit... 2024-09-16T07:12:23.827 INFO:tasks.workunit:Pulling workunits from ref ae7f37330c1a72493684eb1fcaed6f2a01195bb6 2024-09-16T07:12:23.828 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-16T07:12:23.829 INFO:tasks.workunit:timeout=3h 2024-09-16T07:12:23.829 INFO:tasks.workunit:cleanup=True 2024-09-16T07:12:23.829 DEBUG:teuthology.orchestra.run.smithi162:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-16T07:12:23.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T07:12:23.837 INFO:teuthology.orchestra.run.smithi162.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T07:12:23.837 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T07:12:23.885 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-16T07:12:23.885 DEBUG:teuthology.orchestra.run.smithi162:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-16T07:12:23.892 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ae7f37330c1a72493684eb1fcaed6f2a01195bb6 2024-09-16T07:12:23.900 INFO:tasks.workunit.client.0.smithi162.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-16T07:13:20.743 INFO:tasks.workunit.client.0.smithi162.stderr:Note: switching to 'ae7f37330c1a72493684eb1fcaed6f2a01195bb6'. 2024-09-16T07:13:20.743 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-09-16T07:13:20.744 INFO:tasks.workunit.client.0.smithi162.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-16T07:13:20.744 INFO:tasks.workunit.client.0.smithi162.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-16T07:13:20.744 INFO:tasks.workunit.client.0.smithi162.stderr:state without impacting any branches by switching back to a branch. 2024-09-16T07:13:20.744 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-09-16T07:13:20.744 INFO:tasks.workunit.client.0.smithi162.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-16T07:13:20.744 INFO:tasks.workunit.client.0.smithi162.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-16T07:13:20.744 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-09-16T07:13:20.744 INFO:tasks.workunit.client.0.smithi162.stderr: git switch -c 2024-09-16T07:13:20.744 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-09-16T07:13:20.744 INFO:tasks.workunit.client.0.smithi162.stderr:Or undo this operation with: 2024-09-16T07:13:20.744 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-09-16T07:13:20.744 INFO:tasks.workunit.client.0.smithi162.stderr: git switch - 2024-09-16T07:13:20.744 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-09-16T07:13:20.745 INFO:tasks.workunit.client.0.smithi162.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-16T07:13:20.745 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-09-16T07:13:20.745 INFO:tasks.workunit.client.0.smithi162.stderr:HEAD is now at ae7f37330c1 Merge pull request #59795 from zdover23/wip-doc-2024-09-15-backport-59613-to-reef 2024-09-16T07:13:20.749 DEBUG:teuthology.orchestra.run.smithi162:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-16T07:13:20.799 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-16T07:13:20.799 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-16T07:13:20.805 INFO:tasks.workunit:Running workunits matching erasure-code on client.0... 2024-09-16T07:13:20.806 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code-plugins.sh... 2024-09-16T07:13:20.806 DEBUG:teuthology.orchestra.run.smithi162:workunit test erasure-code/test-erasure-code-plugins.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh 2024-09-16T07:13:20.821 INFO:tasks.workunit.client.0.smithi162.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-09-16T07:13:20.821 INFO:tasks.workunit.client.0.smithi162.stderr:++ TIMEOUT=300 2024-09-16T07:13:20.821 INFO:tasks.workunit.client.0.smithi162.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-09-16T07:13:20.822 INFO:tasks.workunit.client.0.smithi162.stderr:++ MAX_TIMEOUT=15 2024-09-16T07:13:20.822 INFO:tasks.workunit.client.0.smithi162.stderr:++ PG_NUM=4 2024-09-16T07:13:20.822 INFO:tasks.workunit.client.0.smithi162.stderr:++ TMPDIR=/tmp 2024-09-16T07:13:20.822 INFO:tasks.workunit.client.0.smithi162.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-09-16T07:13:20.822 INFO:tasks.workunit.client.0.smithi162.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-09-16T07:13:20.822 INFO:tasks.workunit.client.0.smithi162.stderr:++ type xmlstarlet 2024-09-16T07:13:20.823 INFO:tasks.workunit.client.0.smithi162.stderr:++ XMLSTARLET=xmlstarlet 2024-09-16T07:13:20.823 INFO:tasks.workunit.client.0.smithi162.stderr:+++ uname 2024-09-16T07:13:20.824 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' Linux = FreeBSD ']' 2024-09-16T07:13:20.824 INFO:tasks.workunit.client.0.smithi162.stderr:++ SED=sed 2024-09-16T07:13:20.824 INFO:tasks.workunit.client.0.smithi162.stderr:++ AWK=awk 2024-09-16T07:13:20.824 INFO:tasks.workunit.client.0.smithi162.stderr:+++ stty -a 2024-09-16T07:13:20.824 INFO:tasks.workunit.client.0.smithi162.stderr:+++ head -1 2024-09-16T07:13:20.825 INFO:tasks.workunit.client.0.smithi162.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-09-16T07:13:20.833 INFO:tasks.workunit.client.0.smithi162.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T07:13:20.834 INFO:tasks.workunit.client.0.smithi162.stderr:++ termwidth= 2024-09-16T07:13:20.834 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-09-16T07:13:20.834 INFO:tasks.workunit.client.0.smithi162.stderr:++ DIFFCOLOPTS='-y ' 2024-09-16T07:13:20.835 INFO:tasks.workunit.client.0.smithi162.stderr:++ KERNCORE=kernel.core_pattern 2024-09-16T07:13:20.835 INFO:tasks.workunit.client.0.smithi162.stderr:++ EXTRA_OPTS= 2024-09-16T07:13:20.845 INFO:tasks.workunit.client.0.smithi162.stderr:++ test '' = TESTS 2024-09-16T07:13:20.846 INFO:tasks.workunit.client.0.smithi162.stderr:++ uname -m 2024-09-16T07:13:20.849 INFO:tasks.workunit.client.0.smithi162.stderr:+ arch=x86_64 2024-09-16T07:13:20.849 INFO:tasks.workunit.client.0.smithi162.stderr:+ case $arch in 2024-09-16T07:13:20.849 INFO:tasks.workunit.client.0.smithi162.stderr:+ legacy_jerasure_plugins=(jerasure_generic jerasure_sse3 jerasure_sse4) 2024-09-16T07:13:20.849 INFO:tasks.workunit.client.0.smithi162.stderr:+ legacy_shec_plugins=(shec_generic shec_sse3 shec_sse4) 2024-09-16T07:13:20.849 INFO:tasks.workunit.client.0.smithi162.stderr:+ plugins=(jerasure shec lrc isa) 2024-09-16T07:13:20.849 INFO:tasks.workunit.client.0.smithi162.stderr:+ main test-erasure-code-plugins 2024-09-16T07:13:20.849 INFO:tasks.workunit.client.0.smithi162.stderr:+ local dir=td/test-erasure-code-plugins 2024-09-16T07:13:20.849 INFO:tasks.workunit.client.0.smithi162.stderr:+ shift 2024-09-16T07:13:20.849 INFO:tasks.workunit.client.0.smithi162.stderr:+ shopt -s -o xtrace 2024-09-16T07:13:20.850 INFO:tasks.workunit.client.0.smithi162.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T07:13:20.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T07:13:20.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T07:13:20.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T07:13:20.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T07:13:20.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T07:13:20.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T07:13:20.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T07:13:20.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T07:13:20.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/test-erasure-code-plugins 2024-09-16T07:13:20.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:26: run: local dir=td/test-erasure-code-plugins 2024-09-16T07:13:20.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:27: run: shift 2024-09-16T07:13:20.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: export CEPH_MON=127.0.0.1:17110 2024-09-16T07:13:20.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: CEPH_MON=127.0.0.1:17110 2024-09-16T07:13:20.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:30: run: export CEPH_ARGS 2024-09-16T07:13:20.851 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: uuidgen 2024-09-16T07:13:20.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: CEPH_ARGS+='--fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none ' 2024-09-16T07:13:20.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:32: run: CEPH_ARGS+='--mon-host=127.0.0.1:17110 ' 2024-09-16T07:13:20.852 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: set 2024-09-16T07:13:20.852 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T07:13:20.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: local 'funcs=TEST_ec_profile_warning 2024-09-16T07:13:20.855 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_preload_no_warning 2024-09-16T07:13:20.855 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_preload_no_warning_default 2024-09-16T07:13:20.856 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_preload_warning' 2024-09-16T07:13:20.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-09-16T07:13:20.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_ec_profile_warning td/test-erasure-code-plugins 2024-09-16T07:13:20.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:92: TEST_ec_profile_warning: local dir=td/test-erasure-code-plugins 2024-09-16T07:13:20.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:94: TEST_ec_profile_warning: setup td/test-erasure-code-plugins 2024-09-16T07:13:20.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-16T07:13:20.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-16T07:13:20.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:13:20.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:13:20.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:13:20.856 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:13:20.856 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:13:20.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:13:20.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:13:20.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:13:20.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:13:20.860 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:13:20.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:13:20.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:13:20.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:13:20.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:13:20.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:13:20.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:13:20.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:13:20.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:13:20.865 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:13:20.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:13:20.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:13:20.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:13:20.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:13:20.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:13:20.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:13:20.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:13:20.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:13:20.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:13:20.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-16T07:13:20.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T07:13:20.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:13:20.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:13:20.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17283 2024-09-16T07:13:20.876 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T07:13:20.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T07:13:20.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T07:13:20.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T07:13:20.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-16T07:13:20.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:95: TEST_ec_profile_warning: run_mon td/test-erasure-code-plugins a 2024-09-16T07:13:20.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-16T07:13:20.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T07:13:20.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T07:13:20.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T07:13:20.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-16T07:13:20.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2024-09-16T07:13:21.424 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T07:13:21.424 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:13:21.424 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:13:21.425 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:13:21.425 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:13:21.425 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:13:21.425 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:13:21.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T07:13:21.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T07:13:21.635 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T07:13:21.635 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:13:21.635 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:13:21.635 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T07:13:21.635 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T07:13:21.636 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:13:21.636 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:13:21.636 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:13:21.637 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:13:21.637 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:13:21.637 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:13:21.638 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:13:21.638 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:13:21.638 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17283/ceph-mon.a.asok config get fsid 2024-09-16T07:13:21.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T07:13:21.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:13:21.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:13:21.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T07:13:21.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T07:13:21.751 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:13:21.751 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:13:21.751 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:13:21.751 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:13:21.751 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:13:21.752 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:13:21.752 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:13:21.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:13:21.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17283/ceph-mon.a.asok config get mon_host 2024-09-16T07:13:21.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:96: TEST_ec_profile_warning: run_mgr td/test-erasure-code-plugins x 2024-09-16T07:13:21.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-16T07:13:21.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T07:13:21.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T07:13:21.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T07:13:21.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-16T07:13:21.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T07:13:22.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T07:13:22.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:13:22.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:13:22.740 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:13:22.740 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:13:22.740 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:13:22.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:13:22.741 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:13:22.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:13:22.857 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: seq 0 2 2024-09-16T07:13:22.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-09-16T07:13:22.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 0 2024-09-16T07:13:22.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-16T07:13:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:13:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T07:13:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:13:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-16T07:13:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-16T07:13:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:13:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:13:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:13:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-16T07:13:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-16T07:13:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:13:22.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:13:22.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-16T07:13:22.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:13:22.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:13:22.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:13:22.865 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:13:22.865 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:13:22.865 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:13:22.866 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:13:22.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:13:22.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:13:22.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:13:22.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:13:22.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-16T07:13:22.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-16T07:13:22.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:13:22.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:13:22.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:13:22.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:13:22.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:13:22.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:13:22.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-16T07:13:22.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:13:22.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=79baf2ec-003a-450a-bdf5-9ca4e34df49d 2024-09-16T07:13:22.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 79baf2ec-003a-450a-bdf5-9ca4e34df49d' 2024-09-16T07:13:22.870 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 79baf2ec-003a-450a-bdf5-9ca4e34df49d 2024-09-16T07:13:22.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:13:22.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCS2udmRnbVNBAA9nQ+vfdwvvqYU0OL2DWiaA== 2024-09-16T07:13:22.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCS2udmRnbVNBAA9nQ+vfdwvvqYU0OL2DWiaA=="}' 2024-09-16T07:13:22.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 79baf2ec-003a-450a-bdf5-9ca4e34df49d -i td/test-erasure-code-plugins/0/new.json 2024-09-16T07:13:23.254 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:13:23.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-16T07:13:23.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCS2udmRnbVNBAA9nQ+vfdwvvqYU0OL2DWiaA== --osd-uuid 79baf2ec-003a-450a-bdf5-9ca4e34df49d 2024-09-16T07:13:23.316 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:13:23.312+0000 7ffafe47e880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:13:23.347 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:13:23.344+0000 7ffafe47e880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:13:23.349 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:13:23.344+0000 7ffafe47e880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:13:23.350 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:13:23.344+0000 7ffafe47e880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:13:23.350 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:13:23.344+0000 7ffafe47e880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:13:23.350 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:13:23.344+0000 7ffafe47e880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:13:23.448 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:13:23.444+0000 7ffafe47e880 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-16T07:13:29.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-16T07:13:29.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:13:29.076 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T07:13:29.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T07:13:29.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:13:29.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T07:13:29.312 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T07:13:29.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:13:29.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:13:29.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:13:29.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:13:29.346 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:13:29.340+0000 7fcd89970880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:13:29.348 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:13:29.344+0000 7fcd89970880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:13:29.351 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:13:29.348+0000 7fcd89970880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:13:29.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T07:13:29.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:13:29.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T07:13:29.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:13:29.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:13:29.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:13:29.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:13:29.536 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:13:29.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:13:29.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:13:29.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:13:29.893 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:13:29.888+0000 7fcd89970880 -1 Falling back to public interface 2024-09-16T07:13:30.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:13:30.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:13:30.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:13:30.756 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:13:30.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:13:30.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:13:30.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:13:31.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:13:31.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:13:31.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:13:31.979 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:13:31.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:13:31.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:13:32.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:13:32.702 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:13:32.696+0000 7fcd89970880 -1 osd.0 0 log_to_monitors true 2024-09-16T07:13:33.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:13:33.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:13:33.408 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:13:33.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:13:33.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:13:33.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:13:34.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:13:35.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:13:35.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:13:35.002 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:13:35.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:13:35.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:13:35.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:13:35.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:13:36.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:13:36.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:13:36.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:13:36.509 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:13:36.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:13:36.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:13:36.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:13:37.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:13:37.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:13:37.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:13:37.925 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:13:37.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:13:37.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:13:38.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:13:39.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:13:39.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:13:39.455 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:13:39.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:13:39.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:13:39.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:13:39.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:13:40.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:13:40.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:13:40.884 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:13:40.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:13:40.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:13:40.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:13:41.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:13:42.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:13:42.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:13:42.433 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:13:42.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:13:42.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:13:42.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:13:42.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:13:43.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:13:43.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:13:43.892 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:13:43.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:13:43.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:13:43.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:13:44.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:13:45.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:13:45.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:13:45.444 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:13:45.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:13:45.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:13:45.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:13:45.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:13:46.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:13:46.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:13:46.983 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:13:46.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:13:46.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:13:46.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:13:47.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:13:48.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:13:48.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:13:48.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:13:48.464 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:13:48.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:13:48.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:13:49.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:13:50.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:13:50.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:13:50.360 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:13:50.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:13:50.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:13:50.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:13:51.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:13:52.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:13:52.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:13:52.545 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:13:52.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:13:52.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:13:52.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:13:53.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:13:54.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:13:54.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:13:54.073 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:13:54.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:13:54.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:13:54.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:13:55.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:13:56.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:13:56.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:13:56.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:13:56.789 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:13:56.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:13:56.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:13:57.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:13:58.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:13:58.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:13:58.603 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-16T07:13:58.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T07:13:58.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:13:58.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:13:59.040 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2714894167,v1:127.0.0.1:6801/2714894167] [v2:127.0.0.1:6802/2714894167,v1:127.0.0.1:6803/2714894167] exists,up 79baf2ec-003a-450a-bdf5-9ca4e34df49d 2024-09-16T07:13:59.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:13:59.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:13:59.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:13:59.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-09-16T07:13:59.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 1 2024-09-16T07:13:59.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-16T07:13:59.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:13:59.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T07:13:59.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:13:59.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/1 2024-09-16T07:13:59.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-16T07:13:59.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:13:59.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:13:59.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:13:59.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/1' 2024-09-16T07:13:59.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/1/journal' 2024-09-16T07:13:59.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:13:59.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:13:59.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-16T07:13:59.045 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:13:59.045 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:13:59.045 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:13:59.046 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:13:59.046 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:13:59.046 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:13:59.046 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:13:59.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:13:59.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:13:59.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:13:59.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:13:59.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-16T07:13:59.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-16T07:13:59.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:13:59.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:13:59.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:13:59.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:13:59.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:13:59.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:13:59.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/1 2024-09-16T07:13:59.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:13:59.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=de26d826-f36b-43bd-9b66-a28798041ed9 2024-09-16T07:13:59.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 de26d826-f36b-43bd-9b66-a28798041ed9' 2024-09-16T07:13:59.048 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 de26d826-f36b-43bd-9b66-a28798041ed9 2024-09-16T07:13:59.048 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:13:59.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC32udm8coeBBAAcLuM8/z50+tLXZNpG6agzQ== 2024-09-16T07:13:59.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC32udm8coeBBAAcLuM8/z50+tLXZNpG6agzQ=="}' 2024-09-16T07:13:59.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new de26d826-f36b-43bd-9b66-a28798041ed9 -i td/test-erasure-code-plugins/1/new.json 2024-09-16T07:13:59.701 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:13:59.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/1/new.json 2024-09-16T07:13:59.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC32udm8coeBBAAcLuM8/z50+tLXZNpG6agzQ== --osd-uuid de26d826-f36b-43bd-9b66-a28798041ed9 2024-09-16T07:13:59.755 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:13:59.752+0000 7f7853dd1880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:13:59.783 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:13:59.780+0000 7f7853dd1880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:13:59.786 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:13:59.780+0000 7f7853dd1880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:13:59.787 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:13:59.784+0000 7f7853dd1880 -1 bluestore(td/test-erasure-code-plugins/1/block) _read_bdev_label failed to open td/test-erasure-code-plugins/1/block: (2) No such file or directory 2024-09-16T07:13:59.787 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:13:59.784+0000 7f7853dd1880 -1 bluestore(td/test-erasure-code-plugins/1/block) _read_bdev_label failed to open td/test-erasure-code-plugins/1/block: (2) No such file or directory 2024-09-16T07:13:59.787 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:13:59.784+0000 7f7853dd1880 -1 bluestore(td/test-erasure-code-plugins/1/block) _read_bdev_label failed to open td/test-erasure-code-plugins/1/block: (2) No such file or directory 2024-09-16T07:13:59.874 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:13:59.872+0000 7f7853dd1880 -1 bluestore(td/test-erasure-code-plugins/1) _read_fsid unparsable uuid 2024-09-16T07:14:05.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/1/keyring 2024-09-16T07:14:05.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:14:05.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T07:14:05.833 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-16T07:14:05.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:14:06.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T07:14:06.398 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-16T07:14:06.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:14:06.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:14:06.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:14:06.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:14:06.431 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:14:06.428+0000 7fc2c5d60880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:14:06.438 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:14:06.436+0000 7fc2c5d60880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:14:06.442 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:14:06.440+0000 7fc2c5d60880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:14:06.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T07:14:06.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:14:06.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T07:14:06.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:14:06.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:14:06.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:06.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:14:06.885 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:14:06.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:06.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:14:06.996 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:14:06.992+0000 7fc2c5d60880 -1 Falling back to public interface 2024-09-16T07:14:07.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:08.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:08.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:08.301 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:14:08.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:14:08.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:08.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:14:08.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:09.437 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:14:09.432+0000 7fc2c5d60880 -1 osd.1 0 log_to_monitors true 2024-09-16T07:14:09.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:09.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:09.771 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:14:09.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:14:09.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:09.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:14:10.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:11.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:11.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:11.249 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:14:11.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:14:11.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:11.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:14:11.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:12.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:12.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:12.673 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:14:12.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:14:12.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:12.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:14:13.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:14.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:14.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:14.184 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:14:14.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:14:14.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:14.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:14:14.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:15.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:15.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:15.619 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:14:15.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:14:15.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:15.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:14:16.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:17.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:17.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:17.028 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:14:17.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:14:17.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:17.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:14:17.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:18.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:18.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:18.563 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:14:18.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:14:18.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:18.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:14:19.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:20.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:20.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:20.004 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:14:20.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:14:20.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:20.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:14:20.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:21.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:21.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:21.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:14:21.507 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:14:21.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:21.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:14:22.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:23.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:23.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:23.122 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:14:23.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:14:23.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:23.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:14:24.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:25.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:25.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:25.150 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:14:25.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:14:25.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:25.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:14:25.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:26.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:26.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:26.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:14:26.963 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:14:26.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:14:26.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:27.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:28.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:28.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:28.783 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:14:28.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:14:28.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:28.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:14:29.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:30.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:30.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:30.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:14:30.248 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:14:30.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:30.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:14:31.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:32.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:32.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:32.143 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:14:32.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:14:32.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:32.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:14:32.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:33.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:33.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:33.682 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:14:33.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:14:33.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:33.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:14:34.141 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1238639091,v1:127.0.0.1:6811/1238639091] [v2:127.0.0.1:6812/1238639091,v1:127.0.0.1:6813/1238639091] exists,up de26d826-f36b-43bd-9b66-a28798041ed9 2024-09-16T07:14:34.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:14:34.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:14:34.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:14:34.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-09-16T07:14:34.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 2 2024-09-16T07:14:34.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-16T07:14:34.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:14:34.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T07:14:34.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:14:34.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/2 2024-09-16T07:14:34.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-16T07:14:34.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:14:34.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:14:34.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:14:34.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/2' 2024-09-16T07:14:34.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/2/journal' 2024-09-16T07:14:34.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:14:34.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:14:34.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-16T07:14:34.145 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:14:34.145 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:14:34.145 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:14:34.147 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:14:34.147 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:14:34.147 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:14:34.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:14:34.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:14:34.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:14:34.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:14:34.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:14:34.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-16T07:14:34.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-16T07:14:34.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:14:34.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:14:34.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:14:34.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:14:34.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:14:34.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:14:34.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/2 2024-09-16T07:14:34.195 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:14:34.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d7b43292-4a04-4cab-a21c-47c2f53887f6 2024-09-16T07:14:34.197 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 d7b43292-4a04-4cab-a21c-47c2f53887f6 2024-09-16T07:14:34.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d7b43292-4a04-4cab-a21c-47c2f53887f6' 2024-09-16T07:14:34.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:14:34.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDa2udms9M6DRAAli5+zYPAH9ykc8YEtY26yQ== 2024-09-16T07:14:34.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDa2udms9M6DRAAli5+zYPAH9ykc8YEtY26yQ=="}' 2024-09-16T07:14:34.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d7b43292-4a04-4cab-a21c-47c2f53887f6 -i td/test-erasure-code-plugins/2/new.json 2024-09-16T07:14:34.805 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:14:34.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/2/new.json 2024-09-16T07:14:34.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDa2udms9M6DRAAli5+zYPAH9ykc8YEtY26yQ== --osd-uuid d7b43292-4a04-4cab-a21c-47c2f53887f6 2024-09-16T07:14:34.853 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:14:34.848+0000 7f9381c60880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:14:34.857 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:14:34.856+0000 7f9381c60880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:14:34.858 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:14:34.856+0000 7f9381c60880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:14:34.858 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:14:34.856+0000 7f9381c60880 -1 bluestore(td/test-erasure-code-plugins/2/block) _read_bdev_label failed to open td/test-erasure-code-plugins/2/block: (2) No such file or directory 2024-09-16T07:14:34.859 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:14:34.856+0000 7f9381c60880 -1 bluestore(td/test-erasure-code-plugins/2/block) _read_bdev_label failed to open td/test-erasure-code-plugins/2/block: (2) No such file or directory 2024-09-16T07:14:34.859 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:14:34.856+0000 7f9381c60880 -1 bluestore(td/test-erasure-code-plugins/2/block) _read_bdev_label failed to open td/test-erasure-code-plugins/2/block: (2) No such file or directory 2024-09-16T07:14:34.926 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:14:34.924+0000 7f9381c60880 -1 bluestore(td/test-erasure-code-plugins/2) _read_fsid unparsable uuid 2024-09-16T07:14:41.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/2/keyring 2024-09-16T07:14:41.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:14:41.067 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-16T07:14:41.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T07:14:41.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:14:41.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T07:14:41.624 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-16T07:14:41.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:14:41.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:14:41.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:14:41.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:14:41.657 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:14:41.656+0000 7fe80589c880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:14:41.683 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:14:41.680+0000 7fe80589c880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:14:41.685 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:14:41.684+0000 7fe80589c880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:14:42.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T07:14:42.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:14:42.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T07:14:42.078 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:14:42.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:14:42.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:14:42.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:42.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:14:42.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:42.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:14:42.231 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:14:42.228+0000 7fe80589c880 -1 Falling back to public interface 2024-09-16T07:14:42.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:43.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:43.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:43.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:14:43.547 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:14:43.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:43.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:14:44.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:44.653 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:14:44.648+0000 7fe80589c880 -1 osd.2 0 log_to_monitors true 2024-09-16T07:14:45.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:45.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:45.016 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:14:45.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:14:45.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:45.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:14:45.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:46.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:46.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:46.546 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:14:46.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:14:46.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:46.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:14:47.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:48.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:48.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:48.021 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:14:48.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:14:48.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:48.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:14:48.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:49.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:49.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:49.605 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:14:49.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:14:49.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:49.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:14:50.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:51.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:51.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:51.038 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:14:51.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:14:51.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:51.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:14:51.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:52.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:52.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:52.610 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:14:52.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:14:52.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:52.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:14:53.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:54.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:54.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:54.053 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:14:54.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:14:54.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:54.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:14:54.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:55.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:55.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:55.570 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:14:55.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:14:55.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:55.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:14:56.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:57.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:57.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:57.014 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:14:57.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:14:57.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:14:57.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:57.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:14:58.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:14:58.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:14:58.453 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:14:58.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:14:58.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:14:58.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:14:59.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:15:00.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:15:00.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:15:00.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:15:00.333 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:15:00.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:15:00.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:15:01.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:15:02.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:15:02.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:15:02.048 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:15:02.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:15:02.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:15:02.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:15:02.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:15:03.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:15:03.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:15:03.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:15:03.922 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:15:03.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:15:03.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:15:04.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:15:05.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:15:05.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:15:05.434 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:15:05.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:15:05.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:15:05.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:15:06.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:15:07.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:15:07.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:15:07.145 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:15:07.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:15:07.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:15:07.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:15:07.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:15:08.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:15:08.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:15:08.820 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:15:08.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:15:08.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:15:08.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:15:09.314 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1068436974,v1:127.0.0.1:6819/1068436974] [v2:127.0.0.1:6820/1068436974,v1:127.0.0.1:6821/1068436974] exists,up d7b43292-4a04-4cab-a21c-47c2f53887f6 2024-09-16T07:15:09.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:15:09.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:15:09.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:15:09.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:100: TEST_ec_profile_warning: create_rbd_pool 2024-09-16T07:15:09.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T07:15:09.796 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' does not exist 2024-09-16T07:15:09.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T07:15:09.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T07:15:10.390 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-16T07:15:10.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T07:15:11.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T07:15:13.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:101: TEST_ec_profile_warning: wait_for_clean 2024-09-16T07:15:13.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T07:15:13.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T07:15:13.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T07:15:13.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T07:15:13.675 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T07:15:13.676 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T07:15:13.676 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T07:15:13.676 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T07:15:13.677 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T07:15:13.677 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T07:15:13.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T07:15:13.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T07:15:13.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T07:15:13.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T07:15:13.875 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T07:15:14.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T07:15:14.431 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:15:14.431 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T07:15:14.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T07:15:14.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:15:14.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T07:15:14.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836503 2024-09-16T07:15:14.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836503 2024-09-16T07:15:14.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836503' 2024-09-16T07:15:14.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:15:14.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T07:15:14.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672978 2024-09-16T07:15:14.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672978 2024-09-16T07:15:14.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836503 1-42949672978' 2024-09-16T07:15:14.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:15:14.764 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T07:15:14.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509457 2024-09-16T07:15:14.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509457 2024-09-16T07:15:14.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836503 1-42949672978 2-64424509457' 2024-09-16T07:15:14.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:15:14.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836503 2024-09-16T07:15:14.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:15:14.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T07:15:14.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836503 2024-09-16T07:15:14.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:15:14.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836503 2024-09-16T07:15:14.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-16T07:15:14.916 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836503 2024-09-16T07:15:14.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:15:15.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836503 -lt 21474836503 2024-09-16T07:15:15.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:15:15.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672978 2024-09-16T07:15:15.414 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:15:15.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T07:15:15.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672978 2024-09-16T07:15:15.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:15:15.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672978 2024-09-16T07:15:15.419 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672978 2024-09-16T07:15:15.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-09-16T07:15:15.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T07:15:15.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672978 -lt 42949672978 2024-09-16T07:15:15.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:15:15.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509457 2024-09-16T07:15:15.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:15:15.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T07:15:15.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509457 2024-09-16T07:15:15.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:15:15.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509457 2024-09-16T07:15:15.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-16T07:15:15.884 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509457 2024-09-16T07:15:15.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T07:15:16.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509457 -lt 64424509457 2024-09-16T07:15:16.372 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T07:15:16.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:15:16.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:15:16.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T07:15:16.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T07:15:16.880 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T07:15:16.881 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T07:15:16.881 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T07:15:16.881 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T07:15:16.881 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T07:15:16.881 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T07:15:17.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T07:15:17.320 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T07:15:17.321 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:15:17.321 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:15:17.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T07:15:17.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T07:15:17.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T07:15:17.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-09-16T07:15:17.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_generic crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_generic 2024-09-16T07:15:18.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-16T07:15:18.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:15:18.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:15:18.614 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:15:18.614 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:15:18.614 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:15:18.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:15:18.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-16T07:15:18.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-mon.a.asok log flush 2024-09-16T07:15:18.697 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:15:18.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2024-09-16T07:15:18.715 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:15:18.316+0000 7f4fd4d20700 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-09-16T07:15:18.715 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:15:18.592+0000 7f4fd4d20700 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-09-16T07:15:18.715 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:15:18.592+0000 7f4fd4d20700 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-09-16T07:15:18.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-09-16T07:15:18.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse3 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse3 2024-09-16T07:15:19.526 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-16T07:15:19.526 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:15:19.526 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:15:19.527 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:15:19.527 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:15:19.527 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:15:19.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:15:19.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-16T07:15:19.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-mon.a.asok log flush 2024-09-16T07:15:19.616 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:15:19.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2024-09-16T07:15:19.637 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:15:19.008+0000 7f4fd4d20700 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-09-16T07:15:19.637 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:15:19.504+0000 7f4fd4d20700 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-09-16T07:15:19.637 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:15:19.504+0000 7f4fd4d20700 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-09-16T07:15:19.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-09-16T07:15:19.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse4 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse4 2024-09-16T07:15:20.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-16T07:15:20.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:15:20.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:15:20.288 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:15:20.289 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:15:20.289 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:15:20.289 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:15:20.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-16T07:15:20.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-mon.a.asok log flush 2024-09-16T07:15:20.368 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:15:20.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2024-09-16T07:15:20.382 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:15:19.952+0000 7f4fd4d20700 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-09-16T07:15:20.382 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:15:20.268+0000 7f4fd4d20700 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-09-16T07:15:20.382 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:15:20.268+0000 7f4fd4d20700 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-09-16T07:15:20.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-09-16T07:15:20.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_generic crush-failure-domain=osd plugin=shec_generic 2024-09-16T07:15:20.992 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-16T07:15:20.993 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:15:20.993 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:15:20.993 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:15:20.994 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:15:20.994 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:15:20.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:15:20.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-16T07:15:20.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-mon.a.asok log flush 2024-09-16T07:15:21.078 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:15:21.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2024-09-16T07:15:21.097 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:15:20.684+0000 7f4fd4d20700 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-09-16T07:15:21.097 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:15:20.972+0000 7f4fd4d20700 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-09-16T07:15:21.097 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:15:20.972+0000 7f4fd4d20700 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-09-16T07:15:21.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-09-16T07:15:21.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse3 crush-failure-domain=osd plugin=shec_sse3 2024-09-16T07:15:21.734 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-16T07:15:21.734 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:15:21.734 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:15:21.735 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:15:21.735 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:15:21.735 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:15:21.735 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:15:21.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-16T07:15:21.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-mon.a.asok log flush 2024-09-16T07:15:21.813 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:15:21.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2024-09-16T07:15:21.834 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:15:21.432+0000 7f4fd4d20700 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-09-16T07:15:21.834 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:15:21.716+0000 7f4fd4d20700 0 mon.a@0(leader).osd e30 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-09-16T07:15:21.835 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:15:21.716+0000 7f4fd4d20700 0 mon.a@0(leader).osd e30 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-09-16T07:15:21.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-09-16T07:15:21.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse4 crush-failure-domain=osd plugin=shec_sse4 2024-09-16T07:15:22.500 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-16T07:15:22.500 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:15:22.500 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:15:22.501 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:15:22.501 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:15:22.501 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:15:22.501 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:15:22.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-16T07:15:22.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-mon.a.asok log flush 2024-09-16T07:15:22.581 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:15:22.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2024-09-16T07:15:22.601 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:15:22.160+0000 7f4fd4d20700 0 mon.a@0(leader).osd e30 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-09-16T07:15:22.602 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:15:22.476+0000 7f4fd4d20700 0 mon.a@0(leader).osd e31 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-09-16T07:15:22.602 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:15:22.476+0000 7f4fd4d20700 0 mon.a@0(leader).osd e31 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-09-16T07:15:22.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:115: TEST_ec_profile_warning: teardown td/test-erasure-code-plugins 2024-09-16T07:15:22.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:15:22.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:15:22.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:15:22.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:15:22.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:15:22.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:15:22.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:15:22.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:15:22.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:15:22.719 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:15:22.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:15:22.721 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:15:22.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:15:22.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:15:22.724 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:15:22.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:15:22.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:15:22.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:15:22.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:15:22.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:15:22.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:15:22.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:15:22.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:15:22.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:15:22.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:15:22.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:15:22.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:15:22.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:15:22.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-09-16T07:15:22.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning td/test-erasure-code-plugins 2024-09-16T07:15:22.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:58: TEST_preload_no_warning: local dir=td/test-erasure-code-plugins 2024-09-16T07:15:22.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-09-16T07:15:22.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-09-16T07:15:22.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-16T07:15:22.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-16T07:15:22.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:15:22.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:15:22.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:15:22.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:15:22.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:15:22.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:15:22.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:15:22.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:15:22.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:15:22.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:15:22.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:15:22.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:15:22.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:15:22.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:15:22.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:15:22.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:15:22.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:15:22.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:15:22.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:15:22.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:15:22.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:15:22.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:15:22.770 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:15:22.770 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:15:22.770 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:15:22.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:15:22.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:15:22.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:15:22.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-16T07:15:22.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T07:15:22.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:15:22.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:15:22.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17283 2024-09-16T07:15:22.777 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T07:15:22.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T07:15:22.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T07:15:22.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-16T07:15:22.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure 2024-09-16T07:15:22.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-16T07:15:22.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T07:15:22.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T07:15:22.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T07:15:22.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-16T07:15:22.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure 2024-09-16T07:15:23.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T07:15:23.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:15:23.202 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:15:23.202 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:15:23.202 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:15:23.202 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:15:23.203 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:15:23.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure 2024-09-16T07:15:23.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T07:15:23.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T07:15:23.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:15:23.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:15:23.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T07:15:23.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T07:15:23.412 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:15:23.412 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:15:23.412 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:15:23.413 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:15:23.413 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:15:23.413 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:15:23.413 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:15:23.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:15:23.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17283/ceph-mon.a.asok config get fsid 2024-09-16T07:15:23.508 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T07:15:23.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:15:23.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:15:23.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T07:15:23.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T07:15:23.509 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:15:23.510 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:15:23.510 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:15:23.510 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:15:23.510 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:15:23.510 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:15:23.511 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:15:23.511 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:15:23.512 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17283/ceph-mon.a.asok config get mon_host 2024-09-16T07:15:23.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-09-16T07:15:23.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-16T07:15:23.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T07:15:23.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T07:15:23.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T07:15:23.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-16T07:15:23.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T07:15:24.511 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T07:15:24.511 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:15:24.511 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:15:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:15:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:15:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:15:24.512 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:15:24.513 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:15:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:15:24.554 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-09-16T07:15:24.554 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:15:24.554 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:15:24.555 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:15:24.555 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:15:24.555 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:15:24.555 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:15:24.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-09-16T07:15:24.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-mon.a.asok log flush 2024-09-16T07:15:24.638 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:15:24.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure 2024-09-16T07:15:24.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-16T07:15:24.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:15:24.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T07:15:24.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:15:24.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-16T07:15:24.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-16T07:15:24.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:15:24.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:15:24.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:15:24.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-16T07:15:24.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-16T07:15:24.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:15:24.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:15:24.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-16T07:15:24.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:15:24.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:15:24.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:15:24.652 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:15:24.652 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:15:24.652 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:15:24.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:15:24.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:15:24.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:15:24.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:15:24.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:15:24.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-16T07:15:24.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-16T07:15:24.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:15:24.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:15:24.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:15:24.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:15:24.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:15:24.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure 2024-09-16T07:15:24.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-16T07:15:24.654 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:15:24.654 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 1aaa9d8f-7e98-429d-9b31-772533762da9 2024-09-16T07:15:24.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1aaa9d8f-7e98-429d-9b31-772533762da9 2024-09-16T07:15:24.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1aaa9d8f-7e98-429d-9b31-772533762da9' 2024-09-16T07:15:24.655 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:15:24.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAM2+dmJE27KBAA/z+ubGOZiafZurKphQZJ9g== 2024-09-16T07:15:24.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAM2+dmJE27KBAA/z+ubGOZiafZurKphQZJ9g=="}' 2024-09-16T07:15:24.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1aaa9d8f-7e98-429d-9b31-772533762da9 -i td/test-erasure-code-plugins/0/new.json 2024-09-16T07:15:25.072 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:15:25.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-16T07:15:25.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure --mkfs --key AQAM2+dmJE27KBAA/z+ubGOZiafZurKphQZJ9g== --osd-uuid 1aaa9d8f-7e98-429d-9b31-772533762da9 2024-09-16T07:15:25.124 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:15:25.120+0000 7f01ea29a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:15:25.126 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:15:25.124+0000 7f01ea29a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:15:25.129 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:15:25.128+0000 7f01ea29a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:15:25.129 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:15:25.128+0000 7f01ea29a880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:15:25.129 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:15:25.128+0000 7f01ea29a880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:15:25.129 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:15:25.128+0000 7f01ea29a880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:15:25.232 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:15:25.232+0000 7f01ea29a880 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-16T07:15:30.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-16T07:15:30.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:15:30.731 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T07:15:30.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T07:15:30.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:15:31.280 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T07:15:31.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T07:15:31.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure 2024-09-16T07:15:31.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:15:31.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:15:31.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:15:31.315 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:15:31.312+0000 7fb3e7118880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:15:31.344 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:15:31.340+0000 7fb3e7118880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:15:31.345 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:15:31.344+0000 7fb3e7118880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:15:31.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T07:15:31.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:15:31.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T07:15:31.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:15:31.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:15:31.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:15:31.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:15:31.772 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:15:31.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:15:31.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:15:31.908 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:15:31.904+0000 7fb3e7118880 -1 Falling back to public interface 2024-09-16T07:15:32.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:15:33.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:15:33.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:15:33.276 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:15:33.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:15:33.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:15:33.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:15:33.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:15:34.321 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:15:34.320+0000 7fb3e7118880 -1 osd.0 0 log_to_monitors true 2024-09-16T07:15:34.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:15:34.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:15:34.726 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:15:34.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:15:34.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:15:34.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:15:35.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:15:36.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:15:36.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:15:36.314 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:15:36.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:15:36.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:15:36.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:15:36.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:15:37.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:15:37.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:15:37.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:15:37.745 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:15:37.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:15:37.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:15:38.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:15:39.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:15:39.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:15:39.194 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:15:39.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:15:39.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:15:39.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:15:39.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:15:40.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:15:40.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:15:40.663 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:15:40.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:15:40.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:15:40.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:15:41.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:15:42.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:15:42.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:15:42.148 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:15:42.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:15:42.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:15:42.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:15:42.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:15:43.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:15:43.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:15:43.607 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:15:43.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:15:43.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:15:43.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:15:44.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:15:45.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:15:45.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:15:45.169 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:15:45.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:15:45.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:15:45.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:15:45.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:15:46.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:15:46.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:15:46.604 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:15:46.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:15:46.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:15:46.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:15:47.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:15:48.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:15:48.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:15:48.283 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:15:48.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:15:48.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:15:48.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:15:48.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:15:49.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:15:49.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:15:49.736 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:15:49.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:15:49.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:15:49.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:15:50.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:15:51.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:15:51.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:15:51.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:15:51.714 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:15:51.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:15:51.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:15:52.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:15:53.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:15:53.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:15:53.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:15:53.357 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:15:53.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:15:53.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:15:54.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:15:55.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:15:55.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:15:55.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:15:55.439 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:15:55.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:15:55.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:15:55.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:15:56.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:15:56.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:15:56.976 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:15:56.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:15:56.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:15:56.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:15:57.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:15:58.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:15:58.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:15:58.560 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:15:58.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:15:58.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:15:58.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:15:59.032 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3211825806,v1:127.0.0.1:6803/3211825806] [v2:127.0.0.1:6804/3211825806,v1:127.0.0.1:6805/3211825806] exists,up 1aaa9d8f-7e98-429d-9b31-772533762da9 2024-09-16T07:15:59.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:15:59.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:15:59.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:15:59.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-09-16T07:15:59.034 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T07:15:59.034 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T07:15:59.034 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:15:59.034 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:15:59.034 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:15:59.034 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-osd.0.asok 2024-09-16T07:15:59.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-09-16T07:15:59.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-osd.0.asok log flush 2024-09-16T07:15:59.124 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:15:59.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-09-16T07:15:59.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-09-16T07:15:59.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-09-16T07:15:59.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:15:59.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:15:59.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:15:59.141 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:15:59.141 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:15:59.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:15:59.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:15:59.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:15:59.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:15:59.257 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:15:59.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:15:59.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:15:59.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:15:59.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:15:59.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:15:59.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:15:59.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:15:59.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:15:59.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:15:59.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:15:59.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:15:59.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:15:59.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:15:59.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:15:59.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:15:59.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:15:59.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:15:59.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:15:59.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-09-16T07:15:59.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-09-16T07:15:59.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-16T07:15:59.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-16T07:15:59.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:15:59.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:15:59.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:15:59.281 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:15:59.281 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:15:59.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:15:59.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:15:59.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:15:59.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:15:59.285 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:15:59.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:15:59.287 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:15:59.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:15:59.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:15:59.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:15:59.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:15:59.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:15:59.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:15:59.293 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:15:59.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:15:59.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:15:59.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:15:59.298 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:15:59.298 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:15:59.298 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:15:59.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:15:59.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:15:59.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:15:59.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-16T07:15:59.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T07:15:59.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:15:59.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:15:59.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17283 2024-09-16T07:15:59.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T07:15:59.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T07:15:59.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T07:15:59.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-16T07:15:59.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec 2024-09-16T07:15:59.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-16T07:15:59.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T07:15:59.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T07:15:59.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T07:15:59.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-16T07:15:59.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec 2024-09-16T07:15:59.737 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T07:15:59.737 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:15:59.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:15:59.739 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:15:59.739 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:15:59.739 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:15:59.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:15:59.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec 2024-09-16T07:15:59.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T07:15:59.977 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T07:15:59.977 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:15:59.977 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:15:59.977 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T07:15:59.977 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T07:15:59.978 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:15:59.978 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:15:59.978 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:15:59.979 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:15:59.980 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:15:59.980 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:15:59.980 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:15:59.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:15:59.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17283/ceph-mon.a.asok config get fsid 2024-09-16T07:16:00.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T07:16:00.079 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:16:00.079 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:16:00.079 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T07:16:00.079 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T07:16:00.080 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:16:00.080 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:16:00.080 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:16:00.080 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:16:00.081 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:16:00.081 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:16:00.081 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:16:00.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:16:00.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17283/ceph-mon.a.asok config get mon_host 2024-09-16T07:16:00.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-09-16T07:16:00.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-16T07:16:00.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T07:16:00.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T07:16:00.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T07:16:00.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-16T07:16:00.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T07:16:01.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T07:16:01.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:16:01.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:16:01.070 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:16:01.070 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:16:01.071 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:16:01.071 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:16:01.071 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:16:01.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:16:01.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-09-16T07:16:01.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:16:01.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:16:01.106 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:16:01.106 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:16:01.106 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:16:01.106 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:16:01.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-09-16T07:16:01.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-mon.a.asok log flush 2024-09-16T07:16:01.190 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:16:01.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec 2024-09-16T07:16:01.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-16T07:16:01.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:16:01.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T07:16:01.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:16:01.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-16T07:16:01.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-16T07:16:01.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:16:01.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:16:01.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:16:01.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-16T07:16:01.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-16T07:16:01.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:16:01.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:16:01.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-16T07:16:01.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:16:01.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:16:01.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:16:01.206 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:16:01.206 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:16:01.206 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:16:01.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:16:01.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:16:01.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:16:01.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:16:01.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:16:01.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-16T07:16:01.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-16T07:16:01.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:16:01.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:16:01.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:16:01.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:16:01.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:16:01.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec 2024-09-16T07:16:01.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-16T07:16:01.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:16:01.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9adee6d3-b590-4a1a-a0a2-3c94eaeb10b8 2024-09-16T07:16:01.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9adee6d3-b590-4a1a-a0a2-3c94eaeb10b8' 2024-09-16T07:16:01.208 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 9adee6d3-b590-4a1a-a0a2-3c94eaeb10b8 2024-09-16T07:16:01.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:16:01.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAx2+dmnm9UDRAAFVd+niY/Rkqa3Y7akKK9ng== 2024-09-16T07:16:01.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAx2+dmnm9UDRAAFVd+niY/Rkqa3Y7akKK9ng=="}' 2024-09-16T07:16:01.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9adee6d3-b590-4a1a-a0a2-3c94eaeb10b8 -i td/test-erasure-code-plugins/0/new.json 2024-09-16T07:16:01.577 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:16:01.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-16T07:16:01.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec --mkfs --key AQAx2+dmnm9UDRAAFVd+niY/Rkqa3Y7akKK9ng== --osd-uuid 9adee6d3-b590-4a1a-a0a2-3c94eaeb10b8 2024-09-16T07:16:01.626 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:16:01.624+0000 7fe690d5a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:16:01.628 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:16:01.628+0000 7fe690d5a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:16:01.631 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:16:01.628+0000 7fe690d5a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:16:01.631 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:16:01.628+0000 7fe690d5a880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:16:01.631 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:16:01.628+0000 7fe690d5a880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:16:01.631 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:16:01.628+0000 7fe690d5a880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:16:01.734 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:16:01.732+0000 7fe690d5a880 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-16T07:16:07.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-16T07:16:07.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:16:07.791 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T07:16:07.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T07:16:07.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:16:08.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T07:16:08.353 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T07:16:08.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec 2024-09-16T07:16:08.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:16:08.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:16:08.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:16:08.385 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:16:08.383+0000 7f9fb5bcd880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:16:08.415 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:16:08.411+0000 7f9fb5bcd880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:16:08.417 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:16:08.415+0000 7f9fb5bcd880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:16:08.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T07:16:08.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:16:08.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T07:16:08.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:16:08.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:16:08.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:08.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:16:08.921 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:16:08.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:08.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:08.978 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:16:08.975+0000 7f9fb5bcd880 -1 Falling back to public interface 2024-09-16T07:16:09.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:10.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:10.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:10.404 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:16:10.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:16:10.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:10.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:10.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:11.525 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:16:11.523+0000 7f9fb5bcd880 -1 osd.0 0 log_to_monitors true 2024-09-16T07:16:11.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:11.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:11.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:16:11.821 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:16:11.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:11.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:12.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:13.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:13.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:13.421 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:16:13.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:16:13.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:13.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:13.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:14.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:14.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:14.864 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:16:14.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:16:14.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:14.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:15.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:16.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:16.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:16.359 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:16:16.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:16:16.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:16.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:16.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:17.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:17.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:17.797 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:16:17.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:16:17.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:17.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:18.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:19.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:19.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:19.310 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:16:19.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:16:19.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:19.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:19.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:20.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:20.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:20.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:16:20.755 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:16:20.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:20.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:21.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:22.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:22.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:22.237 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:16:22.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:16:22.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:22.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:22.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:23.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:23.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:23.661 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:16:23.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:16:23.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:23.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:24.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:25.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:25.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:25.180 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:16:25.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:16:25.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:25.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:25.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:26.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:26.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:26.710 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:16:26.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:16:26.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:26.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:27.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:28.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:28.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:28.209 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:16:28.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:16:28.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:28.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:29.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:30.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:30.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:30.057 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:16:30.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:16:30.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:30.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:30.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:31.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:31.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:31.646 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:16:31.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:16:31.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:31.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:32.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:33.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:33.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:33.102 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:16:33.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:16:33.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:33.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:33.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:34.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:34.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:34.879 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:16:34.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:16:34.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:34.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:35.331 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1870024376,v1:127.0.0.1:6803/1870024376] [v2:127.0.0.1:6804/1870024376,v1:127.0.0.1:6805/1870024376] exists,up 9adee6d3-b590-4a1a-a0a2-3c94eaeb10b8 2024-09-16T07:16:35.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:16:35.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:16:35.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:16:35.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-09-16T07:16:35.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T07:16:35.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T07:16:35.334 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:16:35.334 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:16:35.334 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:16:35.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-osd.0.asok 2024-09-16T07:16:35.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-09-16T07:16:35.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-osd.0.asok log flush 2024-09-16T07:16:35.429 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:16:35.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-09-16T07:16:35.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-09-16T07:16:35.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-09-16T07:16:35.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:16:35.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:16:35.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:16:35.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:16:35.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:16:35.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:16:35.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:16:35.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:16:35.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:16:35.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:16:35.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:16:35.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:16:35.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:16:35.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:16:35.764 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:16:35.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:16:35.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:16:35.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:16:35.767 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:16:35.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:16:35.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:16:35.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:16:35.782 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:16:35.782 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:16:35.782 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:16:35.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:16:35.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:16:35.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:16:35.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-09-16T07:16:35.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-09-16T07:16:35.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-16T07:16:35.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-16T07:16:35.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:16:35.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:16:35.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:16:35.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:16:35.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:16:35.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:16:35.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:16:35.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:16:35.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:16:35.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:16:35.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:16:35.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:16:35.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:16:35.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:16:35.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:16:35.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:16:35.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:16:35.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:16:35.797 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:16:35.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:16:35.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:16:35.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:16:35.803 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:16:35.803 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:16:35.803 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:16:35.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:16:35.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:16:35.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:16:35.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-16T07:16:35.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T07:16:35.808 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:16:35.808 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:16:35.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17283 2024-09-16T07:16:35.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T07:16:35.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T07:16:35.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T07:16:35.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-16T07:16:35.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=lrc 2024-09-16T07:16:35.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-16T07:16:35.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T07:16:35.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T07:16:35.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T07:16:35.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-16T07:16:35.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=lrc 2024-09-16T07:16:36.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T07:16:36.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:16:36.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:16:36.242 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:16:36.242 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:16:36.242 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:16:36.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:16:36.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=lrc 2024-09-16T07:16:36.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T07:16:36.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T07:16:36.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:16:36.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:16:36.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T07:16:36.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T07:16:36.470 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:16:36.470 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:16:36.470 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:16:36.471 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:16:36.471 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:16:36.471 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:16:36.471 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:16:36.472 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:16:36.472 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17283/ceph-mon.a.asok config get fsid 2024-09-16T07:16:36.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T07:16:36.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:16:36.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:16:36.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T07:16:36.571 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T07:16:36.571 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:16:36.571 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:16:36.571 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:16:36.572 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:16:36.572 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:16:36.572 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:16:36.572 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:16:36.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:16:36.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17283/ceph-mon.a.asok config get mon_host 2024-09-16T07:16:36.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-09-16T07:16:36.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-16T07:16:36.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T07:16:36.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T07:16:36.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T07:16:36.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-16T07:16:36.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T07:16:37.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T07:16:37.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:16:37.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:16:37.561 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:16:37.561 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:16:37.561 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:16:37.561 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:16:37.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:16:37.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:16:37.603 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-09-16T07:16:37.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:16:37.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:16:37.604 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:16:37.605 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:16:37.605 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:16:37.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:16:37.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-09-16T07:16:37.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-mon.a.asok log flush 2024-09-16T07:16:37.689 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:16:37.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=lrc 2024-09-16T07:16:37.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-16T07:16:37.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:16:37.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T07:16:37.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:16:37.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-16T07:16:37.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-16T07:16:37.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:16:37.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:16:37.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:16:37.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-16T07:16:37.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-16T07:16:37.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:16:37.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:16:37.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-16T07:16:37.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:16:37.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:16:37.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:16:37.701 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:16:37.701 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:16:37.701 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:16:37.703 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:16:37.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:16:37.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:16:37.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:16:37.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:16:37.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-16T07:16:37.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-16T07:16:37.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:16:37.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:16:37.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:16:37.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:16:37.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:16:37.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=lrc 2024-09-16T07:16:37.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-16T07:16:37.704 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:16:37.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fad821ee-d7a5-4033-ab6d-b281eb369c41 2024-09-16T07:16:37.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fad821ee-d7a5-4033-ab6d-b281eb369c41' 2024-09-16T07:16:37.704 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 fad821ee-d7a5-4033-ab6d-b281eb369c41 2024-09-16T07:16:37.705 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:16:37.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBV2+dmxl81KxAAiiF4BJFEQP4feJtR/+8IxQ== 2024-09-16T07:16:37.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBV2+dmxl81KxAAiiF4BJFEQP4feJtR/+8IxQ=="}' 2024-09-16T07:16:37.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fad821ee-d7a5-4033-ab6d-b281eb369c41 -i td/test-erasure-code-plugins/0/new.json 2024-09-16T07:16:38.105 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:16:38.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-16T07:16:38.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc --mkfs --key AQBV2+dmxl81KxAAiiF4BJFEQP4feJtR/+8IxQ== --osd-uuid fad821ee-d7a5-4033-ab6d-b281eb369c41 2024-09-16T07:16:38.157 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:16:38.155+0000 7f1dae312880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:16:38.159 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:16:38.155+0000 7f1dae312880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:16:38.161 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:16:38.159+0000 7f1dae312880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:16:38.161 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:16:38.159+0000 7f1dae312880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:16:38.162 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:16:38.159+0000 7f1dae312880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:16:38.162 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:16:38.159+0000 7f1dae312880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:16:38.236 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:16:38.235+0000 7f1dae312880 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-16T07:16:44.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-16T07:16:44.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:16:44.159 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T07:16:44.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T07:16:44.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:16:44.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T07:16:44.734 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T07:16:44.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc 2024-09-16T07:16:44.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:16:44.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:16:44.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:16:44.759 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:16:44.755+0000 7fb2c53ff880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:16:44.761 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:16:44.759+0000 7fb2c53ff880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:16:44.768 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:16:44.759+0000 7fb2c53ff880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:16:45.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T07:16:45.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:16:45.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T07:16:45.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:16:45.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:16:45.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:45.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:16:45.190 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:16:45.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:45.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:45.330 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:16:45.327+0000 7fb2c53ff880 -1 Falling back to public interface 2024-09-16T07:16:45.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:46.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:46.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:46.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:16:46.629 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:16:46.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:46.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:47.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:47.930 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:16:47.927+0000 7fb2c53ff880 -1 osd.0 0 log_to_monitors true 2024-09-16T07:16:48.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:48.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:48.036 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:16:48.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:16:48.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:48.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:48.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:49.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:49.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:49.563 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:16:49.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:16:49.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:49.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:50.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:51.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:51.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:51.031 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:16:51.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:16:51.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:51.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:51.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:52.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:52.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:52.456 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:16:52.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:16:52.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:52.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:52.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:53.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:53.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:53.944 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:16:53.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:16:53.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:53.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:54.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:55.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:55.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:55.377 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:16:55.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:16:55.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:55.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:55.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:56.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:56.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:56.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:16:56.907 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:16:56.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:56.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:57.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:58.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:58.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:58.366 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:16:58.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:16:58.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:58.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:16:58.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:16:59.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:16:59.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:16:59.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:16:59.910 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:16:59.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:16:59.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:00.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:17:01.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:17:01.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:01.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:17:01.475 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:17:01.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:01.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:02.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:17:03.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:17:03.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:03.711 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:17:03.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:17:03.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:03.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:04.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:17:05.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:17:05.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:05.748 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:17:05.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:17:05.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:05.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:06.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:17:07.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:17:07.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:07.711 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:17:07.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:17:07.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:07.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:08.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:17:09.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:17:09.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:09.211 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:17:09.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:17:09.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:09.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:10.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:17:11.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:17:11.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:11.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:17:11.022 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:17:11.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:11.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:11.448 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1373377999,v1:127.0.0.1:6803/1373377999] [v2:127.0.0.1:6804/1373377999,v1:127.0.0.1:6805/1373377999] exists,up fad821ee-d7a5-4033-ab6d-b281eb369c41 2024-09-16T07:17:11.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:17:11.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:17:11.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:17:11.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-09-16T07:17:11.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T07:17:11.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T07:17:11.450 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:17:11.450 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:17:11.450 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:17:11.451 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-osd.0.asok 2024-09-16T07:17:11.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-09-16T07:17:11.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-osd.0.asok log flush 2024-09-16T07:17:11.543 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:17:11.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-09-16T07:17:11.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-09-16T07:17:11.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-09-16T07:17:11.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:17:11.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:17:11.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:17:11.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:17:11.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:17:11.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:17:11.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:17:11.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:17:11.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:17:11.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:17:11.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:17:11.675 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:17:11.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:17:11.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:17:11.678 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:17:11.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:17:11.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:17:11.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:17:11.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:17:11.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:17:11.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:17:11.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:17:11.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:17:11.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:17:11.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:17:11.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:17:11.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:17:11.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:17:11.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-09-16T07:17:11.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-09-16T07:17:11.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-16T07:17:11.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-16T07:17:11.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:17:11.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:17:11.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:17:11.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:17:11.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:17:11.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:17:11.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:17:11.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:17:11.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:17:11.703 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:17:11.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:17:11.705 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:17:11.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:17:11.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:17:11.707 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:17:11.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:17:11.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:17:11.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:17:11.709 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:17:11.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:17:11.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:17:11.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:17:11.715 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:17:11.715 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:17:11.715 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:17:11.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:17:11.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:17:11.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:17:11.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-16T07:17:11.720 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T07:17:11.720 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:17:11.720 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:17:11.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17283 2024-09-16T07:17:11.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T07:17:11.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T07:17:11.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T07:17:11.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-16T07:17:11.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=isa 2024-09-16T07:17:11.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-16T07:17:11.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T07:17:11.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T07:17:11.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T07:17:11.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-16T07:17:11.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=isa 2024-09-16T07:17:12.153 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T07:17:12.153 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:17:12.154 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:17:12.154 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:17:12.154 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:17:12.154 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:17:12.155 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:17:12.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=isa 2024-09-16T07:17:12.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T07:17:12.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T07:17:12.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:17:12.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:17:12.376 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T07:17:12.376 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T07:17:12.377 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:17:12.377 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:17:12.377 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:17:12.377 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:17:12.378 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:17:12.378 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:17:12.378 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:17:12.378 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:17:12.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17283/ceph-mon.a.asok config get fsid 2024-09-16T07:17:12.492 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T07:17:12.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:17:12.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:17:12.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T07:17:12.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T07:17:12.493 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:17:12.494 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:17:12.494 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:17:12.494 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:17:12.494 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:17:12.494 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:17:12.495 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:17:12.495 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:17:12.495 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17283/ceph-mon.a.asok config get mon_host 2024-09-16T07:17:12.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-09-16T07:17:12.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-16T07:17:12.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T07:17:12.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T07:17:12.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T07:17:12.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-16T07:17:12.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T07:17:13.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T07:17:13.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:17:13.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:17:13.494 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:17:13.494 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:17:13.494 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:17:13.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:17:13.495 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:17:13.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:17:13.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-09-16T07:17:13.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:17:13.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:17:13.534 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:17:13.534 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:17:13.534 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:17:13.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:17:13.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-09-16T07:17:13.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-mon.a.asok log flush 2024-09-16T07:17:13.621 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:17:13.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=isa 2024-09-16T07:17:13.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-16T07:17:13.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:17:13.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T07:17:13.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:17:13.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-16T07:17:13.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-16T07:17:13.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:17:13.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:17:13.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:17:13.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-16T07:17:13.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-16T07:17:13.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:17:13.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:17:13.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-16T07:17:13.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:17:13.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:17:13.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:17:13.634 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:17:13.634 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:17:13.634 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:17:13.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:17:13.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:17:13.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:17:13.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:17:13.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:17:13.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-16T07:17:13.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-16T07:17:13.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:17:13.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:17:13.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:17:13.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:17:13.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:17:13.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=isa 2024-09-16T07:17:13.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-16T07:17:13.636 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:17:13.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e62fe0e1-8d4f-4f8b-8942-04eff16e7cf5 2024-09-16T07:17:13.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e62fe0e1-8d4f-4f8b-8942-04eff16e7cf5' 2024-09-16T07:17:13.637 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 e62fe0e1-8d4f-4f8b-8942-04eff16e7cf5 2024-09-16T07:17:13.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:17:13.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB52+dmg1B1JxAAf5sfep8nduaoZJBj/k+l/w== 2024-09-16T07:17:13.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB52+dmg1B1JxAAf5sfep8nduaoZJBj/k+l/w=="}' 2024-09-16T07:17:13.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e62fe0e1-8d4f-4f8b-8942-04eff16e7cf5 -i td/test-erasure-code-plugins/0/new.json 2024-09-16T07:17:14.037 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:17:14.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-16T07:17:14.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa --mkfs --key AQB52+dmg1B1JxAAf5sfep8nduaoZJBj/k+l/w== --osd-uuid e62fe0e1-8d4f-4f8b-8942-04eff16e7cf5 2024-09-16T07:17:14.085 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:17:14.086+0000 7f6d15ea5880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:17:14.088 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:17:14.086+0000 7f6d15ea5880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:17:14.090 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:17:14.090+0000 7f6d15ea5880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:17:14.090 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:17:14.090+0000 7f6d15ea5880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:17:14.091 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:17:14.090+0000 7f6d15ea5880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:17:14.091 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:17:14.090+0000 7f6d15ea5880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:17:14.152 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:17:14.150+0000 7f6d15ea5880 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-16T07:17:20.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-16T07:17:20.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:17:20.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T07:17:20.151 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T07:17:20.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:17:20.709 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T07:17:20.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T07:17:20.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa 2024-09-16T07:17:20.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:17:20.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:17:20.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:17:20.744 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:17:20.746+0000 7f8f41658880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:17:20.770 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:17:20.770+0000 7f8f41658880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:17:20.771 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:17:20.770+0000 7f8f41658880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:17:21.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T07:17:21.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:17:21.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T07:17:21.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:17:21.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:17:21.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:21.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:17:21.153 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:17:21.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:21.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:21.330 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:17:21.330+0000 7f8f41658880 -1 Falling back to public interface 2024-09-16T07:17:21.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:17:22.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:17:22.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:22.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:17:22.530 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:17:22.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:22.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:22.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:17:23.656 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:17:23.654+0000 7f8f41658880 -1 osd.0 0 log_to_monitors true 2024-09-16T07:17:23.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:17:23.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:23.952 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:17:23.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:17:23.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:23.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:24.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:17:25.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:17:25.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:25.496 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:17:25.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:17:25.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:25.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:25.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:17:26.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:17:26.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:26.987 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:17:26.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:17:26.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:26.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:27.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:17:28.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:17:28.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:28.482 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:17:28.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:17:28.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:28.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:28.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:17:29.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:17:29.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:29.897 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:17:29.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:17:29.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:29.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:30.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:17:31.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:17:31.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:31.371 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:17:31.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:17:31.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:31.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:31.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:17:32.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:17:32.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:32.791 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:17:32.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:17:32.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:32.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:33.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:17:34.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:17:34.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:34.415 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:17:34.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:17:34.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:34.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:34.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:17:35.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:17:35.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:35.842 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:17:35.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:17:35.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:35.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:36.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:17:37.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:17:37.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:37.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:17:37.554 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:17:37.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:37.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:38.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:17:39.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:17:39.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:39.745 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:17:39.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:17:39.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:39.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:40.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:17:41.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:17:41.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:41.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:17:41.924 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:17:41.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:41.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:42.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:17:43.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:17:43.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:43.717 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:17:43.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:17:43.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:43.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:44.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:17:45.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:17:45.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:45.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:17:45.597 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:17:45.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:45.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:46.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:17:47.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:17:47.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:47.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:17:47.546 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:17:47.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:47.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:48.085 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/602054636,v1:127.0.0.1:6803/602054636] [v2:127.0.0.1:6804/602054636,v1:127.0.0.1:6805/602054636] exists,up e62fe0e1-8d4f-4f8b-8942-04eff16e7cf5 2024-09-16T07:17:48.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:17:48.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:17:48.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:17:48.086 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-09-16T07:17:48.086 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T07:17:48.086 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T07:17:48.087 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:17:48.087 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:17:48.087 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:17:48.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-osd.0.asok 2024-09-16T07:17:48.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-09-16T07:17:48.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-osd.0.asok log flush 2024-09-16T07:17:48.182 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:17:48.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-09-16T07:17:48.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-09-16T07:17:48.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-09-16T07:17:48.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:17:48.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:17:48.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:17:48.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:17:48.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:17:48.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:17:48.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:17:48.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:17:48.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:17:48.311 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:17:48.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:17:48.313 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:17:48.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:17:48.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:17:48.316 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:17:48.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:17:48.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:17:48.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:17:48.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:17:48.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:17:48.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:17:48.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:17:48.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:17:48.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:17:48.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:17:48.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:17:48.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:17:48.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:17:48.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:72: TEST_preload_no_warning: return 0 2024-09-16T07:17:48.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-09-16T07:17:48.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning_default td/test-erasure-code-plugins 2024-09-16T07:17:48.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:76: TEST_preload_no_warning_default: local dir=td/test-erasure-code-plugins 2024-09-16T07:17:48.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:78: TEST_preload_no_warning_default: setup td/test-erasure-code-plugins 2024-09-16T07:17:48.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-16T07:17:48.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-16T07:17:48.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:17:48.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:17:48.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:17:48.337 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:17:48.337 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:17:48.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:17:48.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:17:48.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:17:48.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:17:48.341 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:17:48.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:17:48.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:17:48.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:17:48.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:17:48.345 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:17:48.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:17:48.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:17:48.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:17:48.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:17:48.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:17:48.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:17:48.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:17:48.352 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:17:48.352 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:17:48.352 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:17:48.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:17:48.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:17:48.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:17:48.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-16T07:17:48.356 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T07:17:48.357 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:17:48.357 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:17:48.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17283 2024-09-16T07:17:48.359 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T07:17:48.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T07:17:48.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T07:17:48.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-16T07:17:48.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:79: TEST_preload_no_warning_default: run_mon td/test-erasure-code-plugins a 2024-09-16T07:17:48.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-16T07:17:48.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T07:17:48.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T07:17:48.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T07:17:48.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-16T07:17:48.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2024-09-16T07:17:48.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T07:17:48.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:17:48.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:17:48.774 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:17:48.774 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:17:48.774 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:17:48.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:17:48.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T07:17:49.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T07:17:49.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T07:17:49.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:17:49.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:17:49.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T07:17:49.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T07:17:49.042 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:17:49.042 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:17:49.042 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:17:49.043 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:17:49.043 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:17:49.043 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:17:49.043 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:17:49.043 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:17:49.043 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17283/ceph-mon.a.asok config get fsid 2024-09-16T07:17:49.142 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T07:17:49.142 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:17:49.142 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:17:49.142 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T07:17:49.142 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T07:17:49.142 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:17:49.142 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:17:49.142 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:17:49.142 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:17:49.142 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:17:49.142 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:17:49.143 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:17:49.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:17:49.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17283/ceph-mon.a.asok config get mon_host 2024-09-16T07:17:49.237 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: get_asok_path mon.a 2024-09-16T07:17:49.237 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:17:49.237 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:17:49.237 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:17:49.237 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:17:49.238 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:17:49.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:17:49.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: CEPH_ARGS= 2024-09-16T07:17:49.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-mon.a.asok log flush 2024-09-16T07:17:49.316 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:17:49.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:81: TEST_preload_no_warning_default: run_mgr td/test-erasure-code-plugins x 2024-09-16T07:17:49.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-16T07:17:49.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T07:17:49.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T07:17:49.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T07:17:49.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-16T07:17:49.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T07:17:50.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T07:17:50.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:17:50.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:17:50.242 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:17:50.242 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:17:50.242 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:17:50.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:17:50.243 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:17:50.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:17:50.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:82: TEST_preload_no_warning_default: run_osd td/test-erasure-code-plugins 0 2024-09-16T07:17:50.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-16T07:17:50.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:17:50.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T07:17:50.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:17:50.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-16T07:17:50.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-16T07:17:50.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:17:50.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:17:50.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:17:50.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-16T07:17:50.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-16T07:17:50.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:17:50.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:17:50.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-16T07:17:50.283 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:17:50.283 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:17:50.283 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:17:50.285 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:17:50.285 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:17:50.285 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:17:50.285 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:17:50.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:17:50.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:17:50.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:17:50.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:17:50.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-16T07:17:50.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-16T07:17:50.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:17:50.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:17:50.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:17:50.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:17:50.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:17:50.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:17:50.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-16T07:17:50.287 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:17:50.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=78082bc1-1678-44a5-a3f3-6e4dd707da73 2024-09-16T07:17:50.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 78082bc1-1678-44a5-a3f3-6e4dd707da73' 2024-09-16T07:17:50.287 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 78082bc1-1678-44a5-a3f3-6e4dd707da73 2024-09-16T07:17:50.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:17:50.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCe2+dmCnJzEhAAj8e5Cl3TiFm6mi/BVno6zA== 2024-09-16T07:17:50.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCe2+dmCnJzEhAAj8e5Cl3TiFm6mi/BVno6zA=="}' 2024-09-16T07:17:50.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 78082bc1-1678-44a5-a3f3-6e4dd707da73 -i td/test-erasure-code-plugins/0/new.json 2024-09-16T07:17:50.673 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:17:50.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-16T07:17:50.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCe2+dmCnJzEhAAj8e5Cl3TiFm6mi/BVno6zA== --osd-uuid 78082bc1-1678-44a5-a3f3-6e4dd707da73 2024-09-16T07:17:50.724 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:17:50.725+0000 7f8888f35880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:17:50.727 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:17:50.725+0000 7f8888f35880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:17:50.729 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:17:50.729+0000 7f8888f35880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:17:50.730 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:17:50.729+0000 7f8888f35880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:17:50.730 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:17:50.729+0000 7f8888f35880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:17:50.731 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:17:50.729+0000 7f8888f35880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:17:50.856 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:17:50.857+0000 7f8888f35880 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-16T07:17:56.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-16T07:17:56.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:17:56.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T07:17:56.641 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T07:17:56.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:17:57.203 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T07:17:57.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T07:17:57.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:17:57.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:17:57.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:17:57.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:17:57.236 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:17:57.233+0000 7fc63d47b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:17:57.238 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:17:57.237+0000 7fc63d47b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:17:57.240 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:17:57.241+0000 7fc63d47b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:17:57.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T07:17:57.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:17:57.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T07:17:57.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:17:57.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:17:57.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:57.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:17:57.617 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:17:57.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:57.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:57.798 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:17:57.797+0000 7fc63d47b880 -1 Falling back to public interface 2024-09-16T07:17:58.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:17:59.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:17:59.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:17:59.018 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:17:59.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:17:59.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:17:59.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:17:59.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:00.369 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:18:00.369+0000 7fc63d47b880 -1 osd.0 0 log_to_monitors true 2024-09-16T07:18:00.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:00.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:00.439 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:18:00.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:18:00.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:00.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:00.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:01.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:01.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:01.962 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:18:01.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:18:01.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:01.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:02.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:03.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:03.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:03.432 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:18:03.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:18:03.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:03.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:03.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:04.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:04.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:04.851 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:18:04.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:18:04.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:04.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:05.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:06.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:06.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:06.421 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:18:06.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:18:06.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:06.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:06.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:07.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:07.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:07.850 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:18:07.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:18:07.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:07.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:08.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:09.283 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:18:09.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:09.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:09.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:18:09.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:09.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:09.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:10.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:10.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:10.777 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:18:10.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:18:10.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:10.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:11.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:12.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:12.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:12.212 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:18:12.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:18:12.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:12.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:12.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:13.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:13.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:13.732 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:18:13.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:18:13.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:13.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:14.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:15.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:15.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:15.292 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:18:15.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:18:15.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:15.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:16.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:17.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:17.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:17.253 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:18:17.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:18:17.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:17.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:18.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:19.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:19.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:19.026 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:18:19.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:18:19.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:19.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:19.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:20.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:20.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:20.977 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:18:20.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:18:20.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:20.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:21.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:22.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:22.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:22.659 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:18:22.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:18:22.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:22.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:23.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:24.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:24.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:24.145 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:18:24.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:18:24.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:24.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:24.612 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/686335696,v1:127.0.0.1:6803/686335696] [v2:127.0.0.1:6804/686335696,v1:127.0.0.1:6805/686335696] exists,up 78082bc1-1678-44a5-a3f3-6e4dd707da73 2024-09-16T07:18:24.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:18:24.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:18:24.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:18:24.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: get_asok_path osd.0 2024-09-16T07:18:24.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T07:18:24.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T07:18:24.615 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:18:24.615 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:18:24.615 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:18:24.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-osd.0.asok 2024-09-16T07:18:24.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: CEPH_ARGS= 2024-09-16T07:18:24.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-osd.0.asok log flush 2024-09-16T07:18:24.710 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:18:24.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:84: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/mon.a.log 2024-09-16T07:18:24.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:85: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/osd.0.log 2024-09-16T07:18:24.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:86: TEST_preload_no_warning_default: teardown td/test-erasure-code-plugins 2024-09-16T07:18:24.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:18:24.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:18:24.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:18:24.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:18:24.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:18:24.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:18:24.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:18:24.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:18:24.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:18:24.840 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:18:24.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:18:24.842 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:18:24.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:18:24.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:18:24.845 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:18:24.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:18:24.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:18:24.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:18:24.848 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:18:24.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:18:24.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:18:24.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:18:24.860 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:18:24.860 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:18:24.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:18:24.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:18:24.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:18:24.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:18:24.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:88: TEST_preload_no_warning_default: return 0 2024-09-16T07:18:24.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-09-16T07:18:24.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_warning td/test-erasure-code-plugins 2024-09-16T07:18:24.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:41: TEST_preload_warning: local dir=td/test-erasure-code-plugins 2024-09-16T07:18:24.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-16T07:18:24.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-16T07:18:24.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-16T07:18:24.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-16T07:18:24.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:18:24.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:18:24.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:18:24.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:18:24.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:18:24.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:18:24.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:18:24.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:18:24.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:18:24.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:18:24.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:18:24.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:18:24.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:18:24.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:18:24.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:18:24.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:18:24.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:18:24.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:18:24.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:18:24.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:18:24.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:18:24.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:18:24.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:18:24.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:18:24.880 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:18:24.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:18:24.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:18:24.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:18:24.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-16T07:18:24.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T07:18:24.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:18:24.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:18:24.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17283 2024-09-16T07:18:24.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T07:18:24.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T07:18:24.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T07:18:24.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-16T07:18:24.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_generic 2024-09-16T07:18:24.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-16T07:18:24.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T07:18:24.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T07:18:24.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T07:18:24.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-16T07:18:24.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_generic 2024-09-16T07:18:25.341 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T07:18:25.341 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:18:25.341 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:18:25.342 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:18:25.342 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:18:25.342 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:18:25.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:18:25.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_generic 2024-09-16T07:18:25.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T07:18:25.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T07:18:25.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:18:25.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:18:25.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T07:18:25.580 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T07:18:25.580 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:18:25.580 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:18:25.580 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:18:25.581 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:18:25.581 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:18:25.581 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:18:25.581 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:18:25.581 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:18:25.581 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17283/ceph-mon.a.asok config get fsid 2024-09-16T07:18:25.679 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T07:18:25.679 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:18:25.679 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:18:25.679 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T07:18:25.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T07:18:25.680 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:18:25.680 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:18:25.680 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:18:25.681 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:18:25.681 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:18:25.681 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:18:25.681 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:18:25.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:18:25.682 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17283/ceph-mon.a.asok config get mon_host 2024-09-16T07:18:25.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-16T07:18:25.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-16T07:18:25.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T07:18:25.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T07:18:25.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T07:18:25.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-16T07:18:25.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T07:18:26.694 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T07:18:26.694 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:18:26.694 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:18:26.695 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:18:26.695 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:18:26.695 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:18:26.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:18:26.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:18:26.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:18:26.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-16T07:18:26.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:18:26.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:18:26.737 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:18:26.737 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:18:26.737 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:18:26.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:18:26.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-16T07:18:26.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-mon.a.asok log flush 2024-09-16T07:18:26.828 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:18:26.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_generic 2024-09-16T07:18:26.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-16T07:18:26.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:18:26.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T07:18:26.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:18:26.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-16T07:18:26.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-16T07:18:26.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:18:26.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:18:26.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:18:26.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-16T07:18:26.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-16T07:18:26.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:18:26.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:18:26.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-16T07:18:26.839 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:18:26.839 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:18:26.839 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:18:26.841 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:18:26.841 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:18:26.841 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:18:26.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:18:26.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:18:26.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:18:26.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:18:26.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:18:26.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-16T07:18:26.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-16T07:18:26.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:18:26.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:18:26.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:18:26.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:18:26.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:18:26.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_generic 2024-09-16T07:18:26.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-16T07:18:26.843 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:18:26.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=75306d1e-12e0-473c-9789-36b8bfdae43e 2024-09-16T07:18:26.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 75306d1e-12e0-473c-9789-36b8bfdae43e' 2024-09-16T07:18:26.845 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 75306d1e-12e0-473c-9789-36b8bfdae43e 2024-09-16T07:18:26.845 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:18:26.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDC2+dm0hqfMxAAsuAEwW+KvIuPX+BdfSMrpA== 2024-09-16T07:18:26.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDC2+dm0hqfMxAAsuAEwW+KvIuPX+BdfSMrpA=="}' 2024-09-16T07:18:26.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 75306d1e-12e0-473c-9789-36b8bfdae43e -i td/test-erasure-code-plugins/0/new.json 2024-09-16T07:18:27.212 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:18:27.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-16T07:18:27.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic --mkfs --key AQDC2+dm0hqfMxAAsuAEwW+KvIuPX+BdfSMrpA== --osd-uuid 75306d1e-12e0-473c-9789-36b8bfdae43e 2024-09-16T07:18:27.260 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:18:27.261+0000 7f3d3f79c880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:18:27.263 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:18:27.261+0000 7f3d3f79c880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:18:27.265 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:18:27.265+0000 7f3d3f79c880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:18:27.266 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:18:27.265+0000 7f3d3f79c880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:18:27.266 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:18:27.265+0000 7f3d3f79c880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:18:27.266 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:18:27.265+0000 7f3d3f79c880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:18:27.347 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:18:27.345+0000 7f3d3f79c880 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-16T07:18:33.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-16T07:18:33.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:18:33.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T07:18:33.243 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T07:18:33.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:18:33.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T07:18:33.811 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T07:18:33.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic 2024-09-16T07:18:33.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:18:33.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:18:33.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:18:33.844 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:18:33.844+0000 7f4d8eb87880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:18:33.872 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:18:33.868+0000 7f4d8eb87880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:18:33.873 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:18:33.872+0000 7f4d8eb87880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:18:34.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T07:18:34.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:18:34.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T07:18:34.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:18:34.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:18:34.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:34.294 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:18:34.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:18:34.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:34.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:34.438 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:18:34.436+0000 7f4d8eb87880 -1 Falling back to public interface 2024-09-16T07:18:34.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:35.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:35.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:35.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:18:35.706 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:18:35.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:35.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:36.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:37.030 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:18:37.028+0000 7f4d8eb87880 -1 osd.0 0 log_to_monitors true 2024-09-16T07:18:37.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:37.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:37.128 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:18:37.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:18:37.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:37.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:37.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:38.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:38.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:38.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:18:38.726 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:18:38.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:38.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:39.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:40.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:40.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:40.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:18:40.182 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:18:40.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:40.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:40.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:41.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:41.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:41.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:18:41.706 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:18:41.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:41.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:42.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:43.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:43.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:43.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:18:43.165 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:18:43.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:43.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:43.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:44.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:44.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:44.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:18:44.750 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:18:44.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:44.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:45.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:46.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:46.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:46.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:18:46.157 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:18:46.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:46.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:46.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:47.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:47.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:47.575 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:18:47.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:18:47.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:47.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:48.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:49.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:49.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:49.149 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:18:49.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:18:49.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:49.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:49.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:50.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:50.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:50.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:18:50.732 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:18:50.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:50.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:51.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:52.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:52.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:52.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:18:52.501 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:18:52.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:52.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:53.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:54.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:54.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:54.597 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:18:54.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:18:54.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:54.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:55.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:56.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:56.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:56.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:18:56.296 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:18:56.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:56.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:57.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:18:58.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:18:58.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:18:58.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:18:58.237 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:18:58.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:18:58.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:18:59.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:00.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:00.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:00.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:19:00.367 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:19:00.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:00.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:00.851 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4089688871,v1:127.0.0.1:6803/4089688871] [v2:127.0.0.1:6804/4089688871,v1:127.0.0.1:6805/4089688871] exists,up 75306d1e-12e0-473c-9789-36b8bfdae43e 2024-09-16T07:19:00.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:19:00.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:19:00.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:19:00.852 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-16T07:19:00.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T07:19:00.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T07:19:00.853 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:19:00.853 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:19:00.853 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:19:00.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-osd.0.asok 2024-09-16T07:19:00.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-16T07:19:00.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-osd.0.asok log flush 2024-09-16T07:19:00.935 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:19:00.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2024-09-16T07:19:00.948 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:18:25.385+0000 7f9a8f48dd40 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-16T07:19:00.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/osd.0.log 2024-09-16T07:19:00.951 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:18:34.440+0000 7f4d8eb87880 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-16T07:19:00.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-16T07:19:00.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:19:00.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:19:00.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:19:00.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:19:00.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:19:00.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:19:00.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:19:00.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:19:01.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:19:01.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:19:01.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:19:01.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:19:01.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:19:01.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:19:01.071 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:19:01.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:19:01.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:19:01.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:19:01.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:19:01.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:19:01.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:19:01.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:19:01.089 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:19:01.089 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:19:01.089 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:19:01.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:19:01.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:19:01.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:19:01.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-16T07:19:01.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-16T07:19:01.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-16T07:19:01.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-16T07:19:01.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:19:01.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:19:01.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:19:01.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:19:01.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:19:01.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:19:01.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:19:01.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:19:01.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:19:01.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:19:01.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:19:01.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:19:01.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:19:01.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:19:01.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:19:01.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:19:01.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:19:01.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:19:01.103 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:19:01.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:19:01.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:19:01.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:19:01.109 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:19:01.109 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:19:01.109 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:19:01.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:19:01.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:19:01.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:19:01.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-16T07:19:01.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T07:19:01.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:19:01.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:19:01.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17283 2024-09-16T07:19:01.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T07:19:01.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T07:19:01.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T07:19:01.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-16T07:19:01.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse3 2024-09-16T07:19:01.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-16T07:19:01.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T07:19:01.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T07:19:01.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T07:19:01.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-16T07:19:01.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse3 2024-09-16T07:19:01.537 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T07:19:01.537 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:19:01.537 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:19:01.538 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:19:01.538 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:19:01.538 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:19:01.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:19:01.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse3 2024-09-16T07:19:01.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T07:19:01.797 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T07:19:01.797 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:19:01.797 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:19:01.797 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T07:19:01.797 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T07:19:01.798 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:19:01.798 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:19:01.798 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:19:01.799 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:19:01.799 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:19:01.799 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:19:01.799 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:19:01.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:19:01.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17283/ceph-mon.a.asok config get fsid 2024-09-16T07:19:01.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T07:19:01.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:19:01.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:19:01.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T07:19:01.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T07:19:01.909 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:19:01.910 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:19:01.910 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:19:01.910 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:19:01.910 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:19:01.910 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:19:01.910 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:19:01.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:19:01.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17283/ceph-mon.a.asok config get mon_host 2024-09-16T07:19:02.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-16T07:19:02.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-16T07:19:02.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T07:19:02.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T07:19:02.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T07:19:02.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-16T07:19:02.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T07:19:02.902 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T07:19:02.902 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:19:02.902 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:19:02.903 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:19:02.903 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:19:02.903 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:19:02.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:19:02.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:19:02.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:19:02.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-16T07:19:02.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:19:02.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:19:02.939 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:19:02.940 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:19:02.940 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:19:02.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:19:02.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-16T07:19:02.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-mon.a.asok log flush 2024-09-16T07:19:03.026 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:19:03.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse3 2024-09-16T07:19:03.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-16T07:19:03.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:19:03.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T07:19:03.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:19:03.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-16T07:19:03.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-16T07:19:03.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:19:03.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:19:03.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:19:03.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-16T07:19:03.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-16T07:19:03.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:19:03.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:19:03.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-16T07:19:03.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:19:03.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:19:03.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:19:03.039 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:19:03.039 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:19:03.039 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:19:03.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:19:03.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:19:03.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:19:03.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:19:03.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:19:03.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-16T07:19:03.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-16T07:19:03.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:19:03.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:19:03.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:19:03.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:19:03.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:19:03.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse3 2024-09-16T07:19:03.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-16T07:19:03.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:19:03.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c097850b-6d37-4400-9d9d-3b0b9e3b6dbb 2024-09-16T07:19:03.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c097850b-6d37-4400-9d9d-3b0b9e3b6dbb' 2024-09-16T07:19:03.041 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 c097850b-6d37-4400-9d9d-3b0b9e3b6dbb 2024-09-16T07:19:03.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:19:03.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDn2+dmf/8CBBAAVW/GQSs26f4zovvtctQ9Dw== 2024-09-16T07:19:03.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDn2+dmf/8CBBAAVW/GQSs26f4zovvtctQ9Dw=="}' 2024-09-16T07:19:03.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c097850b-6d37-4400-9d9d-3b0b9e3b6dbb -i td/test-erasure-code-plugins/0/new.json 2024-09-16T07:19:03.420 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:19:03.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-16T07:19:03.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 --mkfs --key AQDn2+dmf/8CBBAAVW/GQSs26f4zovvtctQ9Dw== --osd-uuid c097850b-6d37-4400-9d9d-3b0b9e3b6dbb 2024-09-16T07:19:03.466 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:19:03.464+0000 7f98037ab880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:19:03.469 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:19:03.468+0000 7f98037ab880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:19:03.472 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:19:03.472+0000 7f98037ab880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:19:03.473 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:19:03.472+0000 7f98037ab880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:19:03.473 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:19:03.472+0000 7f98037ab880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:19:03.473 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:19:03.472+0000 7f98037ab880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:19:03.530 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:19:03.528+0000 7f98037ab880 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-16T07:19:09.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-16T07:19:09.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:19:09.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T07:19:09.367 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T07:19:09.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:19:09.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T07:19:09.929 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T07:19:09.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:19:09.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:19:09.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 2024-09-16T07:19:09.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:19:09.960 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:19:09.956+0000 7f2a1dda5880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:19:09.980 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:19:09.980+0000 7f2a1dda5880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:19:09.982 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:19:09.980+0000 7f2a1dda5880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:19:10.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T07:19:10.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:19:10.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T07:19:10.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:19:10.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:19:10.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:10.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:19:10.415 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:19:10.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:10.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:10.555 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:19:10.552+0000 7f2a1dda5880 -1 Falling back to public interface 2024-09-16T07:19:10.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:11.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:11.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:11.828 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:19:11.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:19:11.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:11.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:12.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:13.119 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:19:13.116+0000 7f2a1dda5880 -1 osd.0 0 log_to_monitors true 2024-09-16T07:19:13.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:13.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:13.236 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:19:13.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:19:13.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:13.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:13.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:14.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:14.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:14.808 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:19:14.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:19:14.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:14.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:15.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:16.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:16.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:16.249 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:19:16.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:19:16.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:16.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:16.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:17.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:17.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:17.677 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:19:17.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:19:17.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:17.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:18.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:19.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:19.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:19.162 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:19:19.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:19:19.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:19.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:19.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:20.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:20.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:20.578 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:19:20.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:19:20.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:20.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:21.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:22.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:22.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:22.061 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:19:22.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:19:22.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:22.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:22.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:23.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:23.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:23.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:19:23.504 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:19:23.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:23.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:23.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:24.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:24.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:24.959 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:19:24.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:19:24.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:24.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:25.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:26.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:26.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:26.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:19:26.527 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:19:26.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:26.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:27.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:28.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:28.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:28.136 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:19:28.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:19:28.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:28.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:29.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:30.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:30.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:30.060 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:19:30.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:19:30.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:30.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:30.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:31.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:31.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:31.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:19:31.577 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:19:31.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:31.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:32.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:33.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:33.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:33.142 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:19:33.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:19:33.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:33.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:33.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:34.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:34.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:34.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:19:34.712 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:19:34.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:34.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:35.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:36.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:36.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:36.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:19:36.458 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:19:36.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:36.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:36.981 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/265841897,v1:127.0.0.1:6803/265841897] [v2:127.0.0.1:6804/265841897,v1:127.0.0.1:6805/265841897] exists,up c097850b-6d37-4400-9d9d-3b0b9e3b6dbb 2024-09-16T07:19:36.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:19:36.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:19:36.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:19:36.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-16T07:19:36.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T07:19:36.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T07:19:36.983 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:19:36.983 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:19:36.983 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:19:36.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-osd.0.asok 2024-09-16T07:19:36.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-16T07:19:36.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-osd.0.asok log flush 2024-09-16T07:19:37.075 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:19:37.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2024-09-16T07:19:37.088 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:19:01.576+0000 7f90a5759d40 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-16T07:19:37.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/osd.0.log 2024-09-16T07:19:37.092 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:19:10.552+0000 7f2a1dda5880 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-16T07:19:37.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-16T07:19:37.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:19:37.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:19:37.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:19:37.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:19:37.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:19:37.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:19:37.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:19:37.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:19:37.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:19:37.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:19:37.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:19:37.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:19:37.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:19:37.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:19:37.213 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:19:37.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:19:37.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:19:37.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:19:37.215 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:19:37.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:19:37.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:19:37.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:19:37.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:19:37.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:19:37.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:19:37.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:19:37.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:19:37.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:19:37.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-16T07:19:37.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-16T07:19:37.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-16T07:19:37.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-16T07:19:37.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:19:37.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:19:37.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:19:37.234 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:19:37.234 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:19:37.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:19:37.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:19:37.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:19:37.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:19:37.240 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:19:37.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:19:37.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:19:37.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:19:37.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:19:37.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:19:37.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:19:37.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:19:37.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:19:37.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:19:37.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:19:37.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:19:37.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:19:37.250 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:19:37.250 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:19:37.251 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:19:37.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:19:37.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:19:37.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:19:37.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-16T07:19:37.254 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T07:19:37.255 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:19:37.255 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:19:37.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17283 2024-09-16T07:19:37.256 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T07:19:37.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T07:19:37.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T07:19:37.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-16T07:19:37.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse4 2024-09-16T07:19:37.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-16T07:19:37.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T07:19:37.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T07:19:37.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T07:19:37.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-16T07:19:37.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse4 2024-09-16T07:19:37.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T07:19:37.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:19:37.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:19:37.718 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:19:37.718 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:19:37.718 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:19:37.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:19:37.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse4 2024-09-16T07:19:37.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T07:19:37.921 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T07:19:37.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:19:37.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:19:37.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T07:19:37.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T07:19:37.923 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:19:37.923 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:19:37.923 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:19:37.924 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:19:37.924 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:19:37.924 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:19:37.924 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:19:37.924 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:19:37.924 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17283/ceph-mon.a.asok config get fsid 2024-09-16T07:19:38.018 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T07:19:38.018 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:19:38.018 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:19:38.018 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T07:19:38.019 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T07:19:38.020 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:19:38.020 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:19:38.020 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:19:38.020 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:19:38.020 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:19:38.020 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:19:38.021 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:19:38.021 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:19:38.021 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17283/ceph-mon.a.asok config get mon_host 2024-09-16T07:19:38.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-16T07:19:38.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-16T07:19:38.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T07:19:38.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T07:19:38.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T07:19:38.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-16T07:19:38.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T07:19:39.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T07:19:39.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:19:39.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:19:39.016 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:19:39.016 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:19:39.016 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:19:39.017 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:19:39.017 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:19:39.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:19:39.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-16T07:19:39.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:19:39.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:19:39.091 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:19:39.091 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:19:39.091 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:19:39.091 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:19:39.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-16T07:19:39.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-mon.a.asok log flush 2024-09-16T07:19:39.177 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:19:39.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse4 2024-09-16T07:19:39.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-16T07:19:39.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:19:39.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T07:19:39.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:19:39.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-16T07:19:39.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-16T07:19:39.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:19:39.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:19:39.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:19:39.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-16T07:19:39.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-16T07:19:39.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:19:39.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:19:39.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-16T07:19:39.188 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:19:39.188 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:19:39.188 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:19:39.190 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:19:39.190 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:19:39.190 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:19:39.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:19:39.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:19:39.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:19:39.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:19:39.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:19:39.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-16T07:19:39.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-16T07:19:39.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:19:39.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:19:39.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:19:39.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:19:39.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:19:39.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse4 2024-09-16T07:19:39.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-16T07:19:39.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:19:39.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=acef9b05-9081-40a4-8f6e-86ad98860611 2024-09-16T07:19:39.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 acef9b05-9081-40a4-8f6e-86ad98860611' 2024-09-16T07:19:39.193 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 acef9b05-9081-40a4-8f6e-86ad98860611 2024-09-16T07:19:39.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:19:39.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAL3Odmghu/DBAAeMMSbxoM5ljVa8BrH3khkw== 2024-09-16T07:19:39.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAL3Odmghu/DBAAeMMSbxoM5ljVa8BrH3khkw=="}' 2024-09-16T07:19:39.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new acef9b05-9081-40a4-8f6e-86ad98860611 -i td/test-erasure-code-plugins/0/new.json 2024-09-16T07:19:39.565 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:19:39.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-16T07:19:39.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 --mkfs --key AQAL3Odmghu/DBAAeMMSbxoM5ljVa8BrH3khkw== --osd-uuid acef9b05-9081-40a4-8f6e-86ad98860611 2024-09-16T07:19:39.616 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:19:39.615+0000 7effe3b98880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:19:39.619 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:19:39.619+0000 7effe3b98880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:19:39.621 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:19:39.619+0000 7effe3b98880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:19:39.621 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:19:39.619+0000 7effe3b98880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:19:39.622 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:19:39.619+0000 7effe3b98880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:19:39.622 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:19:39.619+0000 7effe3b98880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:19:39.705 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:19:39.703+0000 7effe3b98880 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-16T07:19:45.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-16T07:19:45.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:19:45.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T07:19:45.176 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T07:19:45.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:19:45.739 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T07:19:45.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T07:19:45.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:19:45.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:19:45.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:19:45.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 2024-09-16T07:19:45.772 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:19:45.771+0000 7f8b20386880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:19:45.783 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:19:45.779+0000 7f8b20386880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:19:45.784 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:19:45.783+0000 7f8b20386880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:19:46.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T07:19:46.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:19:46.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T07:19:46.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:19:46.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:19:46.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:46.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:19:46.177 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:19:46.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:46.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:46.353 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:19:46.351+0000 7f8b20386880 -1 Falling back to public interface 2024-09-16T07:19:46.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:47.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:47.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:47.627 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:19:47.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:19:47.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:47.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:48.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:48.768 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:19:48.767+0000 7f8b20386880 -1 osd.0 0 log_to_monitors true 2024-09-16T07:19:49.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:49.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:49.056 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:19:49.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:19:49.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:49.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:49.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:50.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:50.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:50.480 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:19:50.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:19:50.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:50.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:50.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:51.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:51.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:51.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:19:51.900 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:19:51.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:51.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:52.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:53.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:53.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:53.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:19:53.322 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:19:53.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:53.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:53.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:54.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:54.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:54.843 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:19:54.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:19:54.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:54.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:55.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:56.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:56.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:56.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:19:56.246 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:19:56.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:56.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:56.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:57.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:57.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:57.685 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:19:57.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:19:57.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:57.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:58.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:19:59.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:19:59.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:19:59.155 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:19:59.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:19:59.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:19:59.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:19:59.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:00.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:00.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:00.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:20:00.619 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:20:00.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:00.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:01.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:02.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:02.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:02.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:20:02.182 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:20:02.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:02.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:03.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:04.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:04.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:04.052 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:20:04.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:20:04.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:04.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:04.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:05.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:05.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:05.653 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:20:05.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:20:05.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:05.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:06.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:07.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:07.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:07.349 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:20:07.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:20:07.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:07.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:07.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:08.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:08.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:08.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:20:08.896 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:20:08.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:08.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:09.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:10.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:10.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:10.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:20:10.386 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:20:10.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:10.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:11.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:12.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:12.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:12.230 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:20:12.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:20:12.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:12.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:12.649 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1574682667,v1:127.0.0.1:6803/1574682667] [v2:127.0.0.1:6804/1574682667,v1:127.0.0.1:6805/1574682667] exists,up acef9b05-9081-40a4-8f6e-86ad98860611 2024-09-16T07:20:12.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:20:12.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:20:12.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:20:12.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-16T07:20:12.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T07:20:12.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T07:20:12.651 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:20:12.652 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:20:12.652 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:20:12.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-osd.0.asok 2024-09-16T07:20:12.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-16T07:20:12.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-osd.0.asok log flush 2024-09-16T07:20:12.733 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:20:12.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2024-09-16T07:20:12.746 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:19:37.759+0000 7feb44791d40 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-16T07:20:12.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/osd.0.log 2024-09-16T07:20:12.748 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:19:46.351+0000 7f8b20386880 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-16T07:20:12.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-16T07:20:12.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:20:12.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:20:12.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:20:12.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:20:12.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:20:12.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:20:12.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:20:12.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:20:12.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:20:12.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:20:12.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:20:12.865 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:20:12.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:20:12.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:20:12.868 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:20:12.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:20:12.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:20:12.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:20:12.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:20:12.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:20:12.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:20:12.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:20:12.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:20:12.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:20:12.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:20:12.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:20:12.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:20:12.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:20:12.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-16T07:20:12.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-16T07:20:12.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-16T07:20:12.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-16T07:20:12.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:20:12.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:20:12.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:20:12.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:20:12.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:20:12.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:20:12.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:20:12.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:20:12.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:20:12.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:20:12.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:20:12.895 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:20:12.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:20:12.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:20:12.897 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:20:12.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:20:12.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:20:12.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:20:12.900 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:20:12.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:20:12.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:20:12.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:20:12.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:20:12.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:20:12.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:20:12.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:20:12.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:20:12.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:20:12.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-16T07:20:12.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T07:20:12.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:20:12.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:20:12.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17283 2024-09-16T07:20:12.912 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T07:20:12.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T07:20:12.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T07:20:12.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-16T07:20:12.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_generic 2024-09-16T07:20:12.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-16T07:20:12.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T07:20:12.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T07:20:12.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T07:20:12.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-16T07:20:12.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_generic 2024-09-16T07:20:13.345 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T07:20:13.345 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:20:13.345 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:20:13.346 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:20:13.346 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:20:13.346 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:20:13.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:20:13.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_generic 2024-09-16T07:20:13.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T07:20:13.575 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T07:20:13.575 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:20:13.575 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:20:13.575 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T07:20:13.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T07:20:13.576 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:20:13.576 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:20:13.576 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:20:13.577 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:20:13.578 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:20:13.578 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:20:13.578 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:20:13.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:20:13.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17283/ceph-mon.a.asok config get fsid 2024-09-16T07:20:13.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T07:20:13.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:20:13.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:20:13.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T07:20:13.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T07:20:13.682 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:20:13.682 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:20:13.682 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:20:13.682 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:20:13.682 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:20:13.682 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:20:13.682 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:20:13.682 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:20:13.682 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17283/ceph-mon.a.asok config get mon_host 2024-09-16T07:20:13.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-16T07:20:13.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-16T07:20:13.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T07:20:13.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T07:20:13.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T07:20:13.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-16T07:20:13.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T07:20:14.676 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T07:20:14.676 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:20:14.676 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:20:14.676 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:20:14.676 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:20:14.676 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:20:14.677 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:20:14.677 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:20:14.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:20:14.755 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-16T07:20:14.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:20:14.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:20:14.756 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:20:14.756 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:20:14.757 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:20:14.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:20:14.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-16T07:20:14.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-mon.a.asok log flush 2024-09-16T07:20:14.843 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:20:14.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_generic 2024-09-16T07:20:14.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-16T07:20:14.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:20:14.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T07:20:14.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:20:14.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-16T07:20:14.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-16T07:20:14.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:20:14.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:20:14.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:20:14.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-16T07:20:14.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-16T07:20:14.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:20:14.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:20:14.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-16T07:20:14.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:20:14.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:20:14.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:20:14.857 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:20:14.857 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:20:14.857 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:20:14.858 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:20:14.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:20:14.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:20:14.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:20:14.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:20:14.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-16T07:20:14.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-16T07:20:14.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:20:14.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:20:14.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:20:14.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:20:14.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:20:14.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_generic 2024-09-16T07:20:14.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-16T07:20:14.860 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:20:14.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=de9b8fba-128b-48a7-a828-363370b629bc 2024-09-16T07:20:14.861 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 de9b8fba-128b-48a7-a828-363370b629bc 2024-09-16T07:20:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 de9b8fba-128b-48a7-a828-363370b629bc' 2024-09-16T07:20:14.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:20:14.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAu3OdmbrrINBAAachyLMQuUBnhOwiXL1Dg+g== 2024-09-16T07:20:14.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAu3OdmbrrINBAAachyLMQuUBnhOwiXL1Dg+g=="}' 2024-09-16T07:20:14.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new de9b8fba-128b-48a7-a828-363370b629bc -i td/test-erasure-code-plugins/0/new.json 2024-09-16T07:20:15.245 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:20:15.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-16T07:20:15.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic --mkfs --key AQAu3OdmbrrINBAAachyLMQuUBnhOwiXL1Dg+g== --osd-uuid de9b8fba-128b-48a7-a828-363370b629bc 2024-09-16T07:20:15.290 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:20:15.286+0000 7f74c3a71880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:20:15.294 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:20:15.290+0000 7f74c3a71880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:20:15.295 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:20:15.294+0000 7f74c3a71880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:20:15.297 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:20:15.294+0000 7f74c3a71880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:20:15.297 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:20:15.294+0000 7f74c3a71880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:20:15.297 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:20:15.294+0000 7f74c3a71880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:20:15.385 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:20:15.382+0000 7f74c3a71880 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-16T07:20:21.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-16T07:20:21.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:20:21.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T07:20:21.287 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T07:20:21.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:20:21.866 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T07:20:21.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T07:20:21.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic 2024-09-16T07:20:21.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:20:21.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:20:21.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:20:21.900 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:20:21.898+0000 7f421366f880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:20:21.908 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:20:21.906+0000 7f421366f880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:20:21.909 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:20:21.906+0000 7f421366f880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:20:22.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T07:20:22.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:20:22.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T07:20:22.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:20:22.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:20:22.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:22.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:20:22.278 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:20:22.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:22.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:22.474 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:20:22.470+0000 7f421366f880 -1 Falling back to public interface 2024-09-16T07:20:22.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:23.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:23.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:23.731 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:20:23.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:20:23.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:23.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:24.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:25.061 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:20:25.062+0000 7f421366f880 -1 osd.0 0 log_to_monitors true 2024-09-16T07:20:25.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:25.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:25.152 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:20:25.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:20:25.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:25.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:25.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:26.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:26.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:26.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:20:26.736 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:20:26.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:26.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:27.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:28.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:28.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:28.223 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:20:28.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:20:28.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:28.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:28.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:29.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:29.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:29.742 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:20:29.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:20:29.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:29.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:30.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:31.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:31.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:31.162 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:20:31.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:20:31.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:31.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:31.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:32.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:32.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:32.558 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:20:32.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:20:32.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:32.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:33.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:34.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:34.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:34.138 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:20:34.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:20:34.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:34.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:34.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:35.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:35.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:35.575 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:20:35.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:20:35.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:35.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:36.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:37.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:37.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:37.139 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:20:37.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:20:37.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:37.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:37.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:38.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:38.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:38.783 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:20:38.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:20:38.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:38.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:39.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:40.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:40.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:40.901 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:20:40.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:20:40.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:40.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:41.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:42.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:42.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:42.998 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:20:42.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:20:42.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:42.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:43.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:44.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:44.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:44.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:20:44.845 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:20:44.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:44.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:45.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:46.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:46.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:46.874 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:20:46.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:20:46.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:46.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:47.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:48.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:48.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:48.427 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:20:48.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:20:48.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:48.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:48.843 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3599421945,v1:127.0.0.1:6803/3599421945] [v2:127.0.0.1:6804/3599421945,v1:127.0.0.1:6805/3599421945] exists,up de9b8fba-128b-48a7-a828-363370b629bc 2024-09-16T07:20:48.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:20:48.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:20:48.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:20:48.844 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-16T07:20:48.844 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T07:20:48.845 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T07:20:48.845 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:20:48.845 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:20:48.845 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:20:48.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-osd.0.asok 2024-09-16T07:20:48.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-16T07:20:48.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-osd.0.asok log flush 2024-09-16T07:20:48.937 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:20:48.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2024-09-16T07:20:48.950 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:20:13.386+0000 7f5eca049d40 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-16T07:20:48.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/osd.0.log 2024-09-16T07:20:48.952 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:20:22.474+0000 7f421366f880 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-16T07:20:48.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-16T07:20:48.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:20:48.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:20:48.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:20:48.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:20:48.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:20:48.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:20:48.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:20:48.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:20:49.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:20:49.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:20:49.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:20:49.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:20:49.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:20:49.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:20:49.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:20:49.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:20:49.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:20:49.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:20:49.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:20:49.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:20:49.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:20:49.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:20:49.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:20:49.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:20:49.091 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:20:49.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:20:49.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:20:49.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:20:49.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-16T07:20:49.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-16T07:20:49.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-16T07:20:49.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-16T07:20:49.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:20:49.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:20:49.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:20:49.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:20:49.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:20:49.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:20:49.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:20:49.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:20:49.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:20:49.099 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:20:49.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:20:49.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:20:49.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:20:49.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:20:49.103 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:20:49.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:20:49.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:20:49.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:20:49.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:20:49.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:20:49.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:20:49.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:20:49.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:20:49.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:20:49.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:20:49.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:20:49.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:20:49.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:20:49.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-16T07:20:49.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T07:20:49.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:20:49.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:20:49.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17283 2024-09-16T07:20:49.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T07:20:49.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T07:20:49.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T07:20:49.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-16T07:20:49.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse3 2024-09-16T07:20:49.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-16T07:20:49.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T07:20:49.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T07:20:49.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T07:20:49.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-16T07:20:49.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse3 2024-09-16T07:20:49.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T07:20:49.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:20:49.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:20:49.518 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:20:49.518 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:20:49.518 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:20:49.518 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:20:49.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse3 2024-09-16T07:20:49.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T07:20:49.747 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T07:20:49.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:20:49.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:20:49.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T07:20:49.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T07:20:49.749 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:20:49.750 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:20:49.750 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:20:49.750 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:20:49.750 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:20:49.750 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:20:49.751 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:20:49.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:20:49.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17283/ceph-mon.a.asok config get fsid 2024-09-16T07:20:49.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T07:20:49.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:20:49.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:20:49.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T07:20:49.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T07:20:49.851 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:20:49.851 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:20:49.851 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:20:49.852 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:20:49.852 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:20:49.852 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:20:49.852 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:20:49.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:20:49.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17283/ceph-mon.a.asok config get mon_host 2024-09-16T07:20:49.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-16T07:20:49.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-16T07:20:49.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T07:20:49.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T07:20:49.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T07:20:49.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-16T07:20:49.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T07:20:50.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T07:20:50.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:20:50.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:20:50.888 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:20:50.888 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:20:50.888 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:20:50.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:20:50.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:20:50.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:20:50.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-16T07:20:50.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:20:50.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:20:50.962 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:20:50.962 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:20:50.962 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:20:50.962 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:20:50.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-16T07:20:50.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-mon.a.asok log flush 2024-09-16T07:20:51.048 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:20:51.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse3 2024-09-16T07:20:51.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-16T07:20:51.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:20:51.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T07:20:51.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:20:51.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-16T07:20:51.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-16T07:20:51.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:20:51.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:20:51.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:20:51.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-16T07:20:51.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-16T07:20:51.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:20:51.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:20:51.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-16T07:20:51.060 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:20:51.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:20:51.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:20:51.061 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:20:51.061 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:20:51.061 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:20:51.062 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:20:51.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:20:51.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:20:51.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:20:51.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:20:51.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-16T07:20:51.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-16T07:20:51.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:20:51.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:20:51.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:20:51.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:20:51.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:20:51.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse3 2024-09-16T07:20:51.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-16T07:20:51.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:20:51.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0880149d-e95e-4db2-9c59-6b84cceac30a 2024-09-16T07:20:51.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0880149d-e95e-4db2-9c59-6b84cceac30a' 2024-09-16T07:20:51.064 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 0880149d-e95e-4db2-9c59-6b84cceac30a 2024-09-16T07:20:51.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:20:51.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBT3OdmWqseBRAA98g06W/qcilp/KJ+TvJ8lw== 2024-09-16T07:20:51.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBT3OdmWqseBRAA98g06W/qcilp/KJ+TvJ8lw=="}' 2024-09-16T07:20:51.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0880149d-e95e-4db2-9c59-6b84cceac30a -i td/test-erasure-code-plugins/0/new.json 2024-09-16T07:20:51.405 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:20:51.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-16T07:20:51.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 --mkfs --key AQBT3OdmWqseBRAA98g06W/qcilp/KJ+TvJ8lw== --osd-uuid 0880149d-e95e-4db2-9c59-6b84cceac30a 2024-09-16T07:20:51.453 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:20:51.454+0000 7f14af8ca880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:20:51.476 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:20:51.478+0000 7f14af8ca880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:20:51.478 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:20:51.478+0000 7f14af8ca880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:20:51.479 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:20:51.478+0000 7f14af8ca880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:20:51.479 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:20:51.478+0000 7f14af8ca880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:20:51.479 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:20:51.478+0000 7f14af8ca880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:20:51.523 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:20:51.522+0000 7f14af8ca880 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-16T07:20:57.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-16T07:20:57.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:20:57.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T07:20:57.503 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T07:20:57.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:20:58.069 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T07:20:58.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T07:20:58.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 2024-09-16T07:20:58.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:20:58.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:20:58.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:20:58.101 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:20:58.101+0000 7fda82df8880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:20:58.125 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:20:58.125+0000 7fda82df8880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:20:58.127 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:20:58.125+0000 7fda82df8880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:20:58.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T07:20:58.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:20:58.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T07:20:58.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:20:58.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:20:58.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:58.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:20:58.541 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:20:58.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:20:58.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:58.694 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:20:58.693+0000 7fda82df8880 -1 Falling back to public interface 2024-09-16T07:20:58.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:20:59.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:20:59.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:20:59.997 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:20:59.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:20:59.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:20:59.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:00.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:01.156 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:21:01.157+0000 7fda82df8880 -1 osd.0 0 log_to_monitors true 2024-09-16T07:21:01.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:01.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:01.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:21:01.420 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:21:01.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:01.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:01.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:02.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:02.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:02.990 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:21:02.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:21:02.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:02.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:03.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:04.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:04.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:04.418 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:21:04.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:21:04.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:04.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:05.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:06.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:06.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:06.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:21:06.003 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:21:06.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:06.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:06.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:07.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:07.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:07.436 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:21:07.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:21:07.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:07.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:07.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:08.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:08.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:08.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:21:08.996 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:21:08.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:08.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:09.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:10.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:10.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:10.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:21:10.436 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:21:10.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:10.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:10.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:11.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:11.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:11.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:21:11.974 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:21:11.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:11.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:12.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:13.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:13.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:13.456 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:21:13.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:21:13.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:13.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:13.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:14.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:14.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:14.962 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:21:14.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:21:14.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:14.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:16.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:17.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:17.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:17.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:21:17.041 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:21:17.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:17.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:18.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:19.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:19.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:19.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:21:19.145 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:21:19.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:19.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:19.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:20.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:20.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:20.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:21:20.566 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:21:20.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:20.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:21.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:22.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:22.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:22.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:21:22.390 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:21:22.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:22.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:22.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:23.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:23.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:23.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:21:23.859 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:21:23.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:23.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:24.367 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1839245717,v1:127.0.0.1:6803/1839245717] [v2:127.0.0.1:6804/1839245717,v1:127.0.0.1:6805/1839245717] exists,up 0880149d-e95e-4db2-9c59-6b84cceac30a 2024-09-16T07:21:24.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:21:24.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:21:24.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:21:24.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-16T07:21:24.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T07:21:24.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T07:21:24.369 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:21:24.369 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:21:24.370 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:21:24.370 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-osd.0.asok 2024-09-16T07:21:24.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-16T07:21:24.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-osd.0.asok log flush 2024-09-16T07:21:24.457 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:21:24.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2024-09-16T07:21:24.467 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:20:49.562+0000 7f26f980bd40 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-16T07:21:24.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/osd.0.log 2024-09-16T07:21:24.469 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:20:58.697+0000 7fda82df8880 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-16T07:21:24.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-16T07:21:24.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:21:24.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:21:24.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:21:24.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:21:24.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:21:24.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:21:24.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:21:24.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:21:24.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:21:24.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:21:24.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:21:24.587 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:21:24.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:21:24.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:21:24.590 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:21:24.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:21:24.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:21:24.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:21:24.593 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:21:24.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:21:24.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:21:24.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:21:24.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:21:24.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:21:24.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:21:24.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:21:24.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:21:24.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:21:24.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-16T07:21:24.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-16T07:21:24.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-16T07:21:24.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-16T07:21:24.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:21:24.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:21:24.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:21:24.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:21:24.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:21:24.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:21:24.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:21:24.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:21:24.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:21:24.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:21:24.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:21:24.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:21:24.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:21:24.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:21:24.619 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:21:24.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:21:24.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:21:24.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:21:24.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:21:24.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:21:24.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:21:24.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:21:24.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:21:24.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:21:24.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:21:24.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:21:24.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:21:24.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:21:24.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-16T07:21:24.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T07:21:24.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:21:24.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:21:24.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17283 2024-09-16T07:21:24.635 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T07:21:24.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T07:21:24.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T07:21:24.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-16T07:21:24.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse4 2024-09-16T07:21:24.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-16T07:21:24.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T07:21:24.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T07:21:24.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T07:21:24.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-16T07:21:24.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse4 2024-09-16T07:21:25.089 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T07:21:25.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:21:25.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:21:25.090 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:21:25.090 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:21:25.090 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:21:25.091 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:21:25.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse4 2024-09-16T07:21:25.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T07:21:25.326 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T07:21:25.326 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:21:25.326 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:21:25.326 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T07:21:25.326 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T07:21:25.326 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:21:25.326 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:21:25.326 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:21:25.326 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:21:25.326 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:21:25.326 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:21:25.327 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:21:25.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:21:25.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17283/ceph-mon.a.asok config get fsid 2024-09-16T07:21:25.425 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T07:21:25.425 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:21:25.425 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:21:25.425 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T07:21:25.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T07:21:25.426 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:21:25.426 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:21:25.426 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:21:25.427 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:21:25.427 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:21:25.427 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:21:25.428 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:21:25.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:21:25.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17283/ceph-mon.a.asok config get mon_host 2024-09-16T07:21:25.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-16T07:21:25.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-16T07:21:25.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T07:21:25.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T07:21:25.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T07:21:25.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-16T07:21:25.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T07:21:26.420 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T07:21:26.420 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:21:26.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:21:26.421 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:21:26.421 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:21:26.421 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:21:26.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:21:26.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:21:26.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:21:26.464 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-16T07:21:26.464 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:21:26.464 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:21:26.464 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:21:26.465 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:21:26.465 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:21:26.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-mon.a.asok 2024-09-16T07:21:26.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-16T07:21:26.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-mon.a.asok log flush 2024-09-16T07:21:26.553 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:21:26.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse4 2024-09-16T07:21:26.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-16T07:21:26.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:21:26.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T07:21:26.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:21:26.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-16T07:21:26.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-16T07:21:26.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:21:26.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:21:26.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:21:26.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-16T07:21:26.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-16T07:21:26.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:21:26.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:21:26.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-16T07:21:26.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:21:26.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:21:26.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:21:26.566 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:21:26.567 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:21:26.567 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:21:26.567 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:21:26.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' 2024-09-16T07:21:26.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:21:26.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:21:26.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:21:26.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-16T07:21:26.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-16T07:21:26.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:21:26.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:21:26.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:21:26.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:21:26.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:21:26.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse4 2024-09-16T07:21:26.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-16T07:21:26.569 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:21:26.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=59734c20-8843-483a-b648-98649f1d61d7 2024-09-16T07:21:26.570 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 59734c20-8843-483a-b648-98649f1d61d7 2024-09-16T07:21:26.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 59734c20-8843-483a-b648-98649f1d61d7' 2024-09-16T07:21:26.571 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:21:26.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB23OdmyLBGIxAAtQ34xNV/qfH871H8NCKIrg== 2024-09-16T07:21:26.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB23OdmyLBGIxAAtQ34xNV/qfH871H8NCKIrg=="}' 2024-09-16T07:21:26.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 59734c20-8843-483a-b648-98649f1d61d7 -i td/test-erasure-code-plugins/0/new.json 2024-09-16T07:21:26.954 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:21:26.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-16T07:21:26.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 --mkfs --key AQB23OdmyLBGIxAAtQ34xNV/qfH871H8NCKIrg== --osd-uuid 59734c20-8843-483a-b648-98649f1d61d7 2024-09-16T07:21:27.006 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:21:27.005+0000 7fd5ee686880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:21:27.009 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:21:27.009+0000 7fd5ee686880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:21:27.010 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:21:27.009+0000 7fd5ee686880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:21:27.012 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:21:27.009+0000 7fd5ee686880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:21:27.012 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:21:27.009+0000 7fd5ee686880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:21:27.012 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:21:27.009+0000 7fd5ee686880 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-16T07:21:27.110 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:21:27.109+0000 7fd5ee686880 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-16T07:21:33.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-16T07:21:33.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:21:33.183 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T07:21:33.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T07:21:33.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:21:33.741 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T07:21:33.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T07:21:33.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d50c9672-a8d8-40c6-a444-f23b0e053933 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 2024-09-16T07:21:33.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:21:33.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:21:33.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:21:33.771 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:21:33.769+0000 7fa710772880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:21:33.794 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:21:33.793+0000 7fa710772880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:21:33.796 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:21:33.797+0000 7fa710772880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:21:34.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T07:21:34.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:21:34.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T07:21:34.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:21:34.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:21:34.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:34.191 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:21:34.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:21:34.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:34.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:34.370 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:21:34.368+0000 7fa710772880 -1 Falling back to public interface 2024-09-16T07:21:34.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:35.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:35.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:35.637 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:21:35.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:21:35.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:35.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:36.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:36.765 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:21:36.764+0000 7fa710772880 -1 osd.0 0 log_to_monitors true 2024-09-16T07:21:37.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:37.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:37.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:21:37.049 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:21:37.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:37.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:37.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:38.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:38.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:38.484 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:21:38.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:21:38.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:38.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:38.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:39.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:39.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:39.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:21:39.907 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:21:39.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:39.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:40.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:41.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:41.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:41.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:21:41.465 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:21:41.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:41.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:41.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:42.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:42.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:42.925 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:21:42.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:21:42.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:42.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:43.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:44.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:44.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:44.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:21:44.533 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:21:44.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:44.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:44.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:45.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:45.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:45.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:21:45.967 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:21:45.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:45.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:46.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:47.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:47.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:47.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:21:47.534 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:21:47.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:47.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:47.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:49.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:49.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:49.001 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:21:49.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:21:49.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:49.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:49.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:50.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:50.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:50.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:21:50.398 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:21:50.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:50.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:51.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:52.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:52.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:52.435 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:21:52.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:21:52.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:52.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:53.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:54.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:54.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:54.019 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:21:54.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:21:54.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:54.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:55.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:56.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:56.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:56.032 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:21:56.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:21:56.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:56.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:56.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:57.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:57.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:57.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:21:57.685 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:21:57.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:57.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:58.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:21:59.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:21:59.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:21:59.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:21:59.349 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:21:59.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:21:59.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:21:59.911 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1243360517,v1:127.0.0.1:6803/1243360517] [v2:127.0.0.1:6804/1243360517,v1:127.0.0.1:6805/1243360517] exists,up 59734c20-8843-483a-b648-98649f1d61d7 2024-09-16T07:21:59.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:21:59.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:21:59.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:21:59.912 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-16T07:21:59.912 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T07:21:59.912 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T07:21:59.913 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:21:59.913 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:21:59.913 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:21:59.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17283/ceph-osd.0.asok 2024-09-16T07:21:59.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-16T07:21:59.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17283/ceph-osd.0.asok log flush 2024-09-16T07:22:00.006 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:22:00.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2024-09-16T07:22:00.020 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:21:25.133+0000 7fb11c16dd40 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-16T07:22:00.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/osd.0.log 2024-09-16T07:22:00.023 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:21:34.368+0000 7fa710772880 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-16T07:22:00.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-16T07:22:00.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:22:00.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:22:00.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:22:00.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:22:00.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:22:00.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:22:00.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:22:00.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:22:00.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:22:00.136 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:22:00.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:22:00.138 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:22:00.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:22:00.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:22:00.141 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:22:00.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:22:00.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:22:00.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:22:00.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:22:00.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:22:00.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:22:00.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:22:00.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:22:00.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:22:00.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:22:00.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:22:00.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:22:00.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:22:00.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:54: TEST_preload_warning: return 0 2024-09-16T07:22:00.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T07:22:00.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/test-erasure-code-plugins 0 2024-09-16T07:22:00.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-16T07:22:00.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T07:22:00.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-16T07:22:00.162 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:22:00.162 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:22:00.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:22:00.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:22:00.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:22:00.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:22:00.167 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:22:00.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:22:00.169 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:22:00.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:22:00.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:22:00.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:22:00.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:22:00.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:22:00.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:22:00.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:22:00.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:22:00.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T07:22:00.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-16T07:22:00.180 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:22:00.180 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:22:00.180 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17283 2024-09-16T07:22:00.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17283 2024-09-16T07:22:00.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:22:00.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:22:00.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T07:22:00.183 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T07:22:00.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T07:22:00.241 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code.sh... 2024-09-16T07:22:00.241 DEBUG:teuthology.orchestra.run.smithi162:workunit test erasure-code/test-erasure-code.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh 2024-09-16T07:22:00.258 INFO:tasks.workunit.client.0.smithi162.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T07:22:00.271 INFO:tasks.workunit.client.0.smithi162.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T07:22:00.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T07:22:00.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T07:22:00.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T07:22:00.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T07:22:00.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T07:22:00.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T07:22:00.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T07:22:00.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T07:22:00.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/test-erasure-code 2024-09-16T07:22:00.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:22: run: local dir=td/test-erasure-code 2024-09-16T07:22:00.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:23: run: shift 2024-09-16T07:22:00.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: export CEPH_MON=127.0.0.1:7101 2024-09-16T07:22:00.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: CEPH_MON=127.0.0.1:7101 2024-09-16T07:22:00.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:26: run: export CEPH_ARGS 2024-09-16T07:22:00.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: uuidgen 2024-09-16T07:22:00.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: CEPH_ARGS+='--fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none ' 2024-09-16T07:22:00.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-16T07:22:00.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:30: run: setup td/test-erasure-code 2024-09-16T07:22:00.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code 2024-09-16T07:22:00.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code 2024-09-16T07:22:00.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2024-09-16T07:22:00.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:22:00.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2024-09-16T07:22:00.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:22:00.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:22:00.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:22:00.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:22:00.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:22:00.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:22:00.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:22:00.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:22:00.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:22:00.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:22:00.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:22:00.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:22:00.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:22:00.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:22:00.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:22:00.284 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:22:00.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:22:00.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:22:00.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2024-09-16T07:22:00.289 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:22:00.289 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:22:00.289 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:22:00.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.46203 2024-09-16T07:22:00.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:22:00.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:22:00.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code 2024-09-16T07:22:00.293 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T07:22:00.293 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:22:00.294 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:22:00.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.46203 2024-09-16T07:22:00.296 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T07:22:00.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T07:22:00.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T07:22:00.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T07:22:00.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code 1' TERM HUP INT 2024-09-16T07:22:00.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:31: run: run_mon td/test-erasure-code a 2024-09-16T07:22:00.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code 2024-09-16T07:22:00.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T07:22:00.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T07:22:00.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T07:22:00.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code/a 2024-09-16T07:22:00.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code/a --run-dir=td/test-erasure-code 2024-09-16T07:22:00.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T07:22:00.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:22:00.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:22:00.734 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:22:00.734 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:22:00.734 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:22:00.734 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:22:00.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code/a '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code/log --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T07:22:00.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T07:22:00.951 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T07:22:00.951 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:22:00.951 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:22:00.951 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T07:22:00.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T07:22:00.952 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:22:00.952 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:22:00.953 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:22:00.953 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:22:00.953 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:22:00.953 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:22:00.954 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46203/ceph-mon.a.asok 2024-09-16T07:22:00.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:22:00.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46203/ceph-mon.a.asok config get fsid 2024-09-16T07:22:01.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T07:22:01.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:22:01.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:22:01.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T07:22:01.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T07:22:01.056 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:22:01.056 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:22:01.056 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:22:01.056 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:22:01.056 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:22:01.056 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:22:01.057 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46203/ceph-mon.a.asok 2024-09-16T07:22:01.057 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:22:01.057 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46203/ceph-mon.a.asok config get mon_host 2024-09-16T07:22:01.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:32: run: run_mgr td/test-erasure-code x 2024-09-16T07:22:01.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code 2024-09-16T07:22:01.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T07:22:01.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T07:22:01.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T07:22:01.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code/x 2024-09-16T07:22:01.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T07:22:02.050 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T07:22:02.050 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:22:02.050 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:22:02.051 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:22:02.051 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:22:02.051 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:22:02.051 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:22:02.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:22:02.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code/x '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:22:02.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: get_asok_path mon.a 2024-09-16T07:22:02.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:22:02.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:22:02.093 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:22:02.093 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:22:02.093 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:22:02.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46203/ceph-mon.a.asok 2024-09-16T07:22:02.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: CEPH_ARGS= 2024-09-16T07:22:02.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: ceph --admin-daemon /tmp/ceph-asok.46203/ceph-mon.a.asok log flush 2024-09-16T07:22:02.176 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:22:02.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:35: run: grep 'load: jerasure.*lrc' td/test-erasure-code/mon.a.log 2024-09-16T07:22:02.186 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:22:00.788+0000 7f785432ad40 0 load: jerasure load: lrc 2024-09-16T07:22:02.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: seq 0 10 2024-09-16T07:22:02.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-16T07:22:02.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 0 2024-09-16T07:22:02.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-16T07:22:02.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:22:02.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T07:22:02.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:22:02.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/0 2024-09-16T07:22:02.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-16T07:22:02.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:22:02.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:22:02.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:22:02.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/0' 2024-09-16T07:22:02.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/0/journal' 2024-09-16T07:22:02.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:22:02.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:22:02.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-16T07:22:02.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:22:02.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:22:02.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:22:02.193 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:22:02.193 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:22:02.193 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:22:02.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:22:02.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:22:02.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:22:02.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:22:02.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:22:02.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-16T07:22:02.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-16T07:22:02.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:22:02.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:22:02.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:22:02.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:22:02.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:22:02.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:22:02.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/0 2024-09-16T07:22:02.194 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:22:02.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=79a72c1d-c78d-4b39-934f-b81f645a4c44 2024-09-16T07:22:02.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 79a72c1d-c78d-4b39-934f-b81f645a4c44' 2024-09-16T07:22:02.195 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 79a72c1d-c78d-4b39-934f-b81f645a4c44 2024-09-16T07:22:02.195 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:22:02.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCa3OdmDaIhDRAA+fmqfnKsJAtuUXjxGzquRQ== 2024-09-16T07:22:02.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCa3OdmDaIhDRAA+fmqfnKsJAtuUXjxGzquRQ=="}' 2024-09-16T07:22:02.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 79a72c1d-c78d-4b39-934f-b81f645a4c44 -i td/test-erasure-code/0/new.json 2024-09-16T07:22:02.611 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:22:02.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/0/new.json 2024-09-16T07:22:02.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCa3OdmDaIhDRAA+fmqfnKsJAtuUXjxGzquRQ== --osd-uuid 79a72c1d-c78d-4b39-934f-b81f645a4c44 2024-09-16T07:22:02.660 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:22:02.656+0000 7f5ca247e880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:22:02.663 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:22:02.660+0000 7f5ca247e880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:22:02.664 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:22:02.664+0000 7f5ca247e880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:22:02.665 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:22:02.664+0000 7f5ca247e880 -1 bluestore(td/test-erasure-code/0/block) _read_bdev_label failed to open td/test-erasure-code/0/block: (2) No such file or directory 2024-09-16T07:22:02.665 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:22:02.664+0000 7f5ca247e880 -1 bluestore(td/test-erasure-code/0/block) _read_bdev_label failed to open td/test-erasure-code/0/block: (2) No such file or directory 2024-09-16T07:22:02.666 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:22:02.664+0000 7f5ca247e880 -1 bluestore(td/test-erasure-code/0/block) _read_bdev_label failed to open td/test-erasure-code/0/block: (2) No such file or directory 2024-09-16T07:22:02.785 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:22:02.784+0000 7f5ca247e880 -1 bluestore(td/test-erasure-code/0) _read_fsid unparsable uuid 2024-09-16T07:22:08.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/0/keyring 2024-09-16T07:22:08.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:22:08.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T07:22:08.831 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T07:22:08.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:22:09.365 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T07:22:09.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T07:22:09.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:22:09.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:22:09.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:22:09.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:22:09.400 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:22:09.400+0000 7f9417297880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:22:09.417 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:22:09.416+0000 7f9417297880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:22:09.420 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:22:09.420+0000 7f9417297880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:22:09.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T07:22:09.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:22:09.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T07:22:09.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:22:09.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:22:09.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:09.917 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:22:09.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:22:09.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:09.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:22:09.998 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:22:09.996+0000 7f9417297880 -1 Falling back to public interface 2024-09-16T07:22:10.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:11.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:11.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:11.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:22:11.353 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:22:11.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:11.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:22:11.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:12.564 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:22:12.560+0000 7f9417297880 -1 osd.0 0 log_to_monitors true 2024-09-16T07:22:12.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:12.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:12.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:22:12.760 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:22:12.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:12.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:22:13.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:14.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:14.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:14.405 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:22:14.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:22:14.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:14.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:22:14.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:15.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:15.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:15.851 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:22:15.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:22:15.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:15.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:22:16.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:17.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:17.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:17.390 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:22:17.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:22:17.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:17.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:22:17.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:18.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:18.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:18.849 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:22:18.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:22:18.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:18.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:22:19.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:20.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:20.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:20.312 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:22:20.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:22:20.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:20.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:22:20.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:21.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:21.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:21.756 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:22:21.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:22:21.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:21.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:22:22.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:23.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:23.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:23.314 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:22:23.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:22:23.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:23.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:22:23.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:24.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:24.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:24.743 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:22:24.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:22:24.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:24.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:22:25.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:26.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:26.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:26.264 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:22:26.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:22:26.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:26.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:22:27.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:28.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:28.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:28.211 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:22:28.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:22:28.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:28.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:22:29.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:30.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:30.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:30.260 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:22:30.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:22:30.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:30.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:22:30.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:31.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:31.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:31.826 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:22:31.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:22:31.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:31.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:22:32.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:33.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:33.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:33.469 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:22:33.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:22:33.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:33.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:22:34.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:35.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:35.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:35.049 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:22:35.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:22:35.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:35.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:22:35.611 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3590159571,v1:127.0.0.1:6803/3590159571] [v2:127.0.0.1:6804/3590159571,v1:127.0.0.1:6805/3590159571] exists,up 79a72c1d-c78d-4b39-934f-b81f645a4c44 2024-09-16T07:22:35.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:22:35.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:22:35.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:22:35.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-16T07:22:35.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 1 2024-09-16T07:22:35.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-16T07:22:35.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:22:35.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T07:22:35.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:22:35.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/1 2024-09-16T07:22:35.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-16T07:22:35.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:22:35.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:22:35.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:22:35.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/1' 2024-09-16T07:22:35.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/1/journal' 2024-09-16T07:22:35.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:22:35.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:22:35.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-16T07:22:35.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:22:35.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:22:35.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:22:35.617 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:22:35.617 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:22:35.617 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:22:35.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:22:35.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:22:35.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:22:35.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:22:35.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:22:35.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-16T07:22:35.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-16T07:22:35.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:22:35.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:22:35.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:22:35.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:22:35.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:22:35.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:22:35.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/1 2024-09-16T07:22:35.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:22:35.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0811deaf-21fd-46c0-843f-bb43fa4bc866 2024-09-16T07:22:35.619 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 0811deaf-21fd-46c0-843f-bb43fa4bc866 2024-09-16T07:22:35.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0811deaf-21fd-46c0-843f-bb43fa4bc866' 2024-09-16T07:22:35.619 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:22:35.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC73OdmMqRqJhAAPJCPlRiek7P1/G5BDAiIOQ== 2024-09-16T07:22:35.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC73OdmMqRqJhAAPJCPlRiek7P1/G5BDAiIOQ=="}' 2024-09-16T07:22:35.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0811deaf-21fd-46c0-843f-bb43fa4bc866 -i td/test-erasure-code/1/new.json 2024-09-16T07:22:36.226 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:22:36.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/1/new.json 2024-09-16T07:22:36.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC73OdmMqRqJhAAPJCPlRiek7P1/G5BDAiIOQ== --osd-uuid 0811deaf-21fd-46c0-843f-bb43fa4bc866 2024-09-16T07:22:36.280 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:22:36.275+0000 7fa5941b6880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:22:36.307 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:22:36.303+0000 7fa5941b6880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:22:36.309 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:22:36.307+0000 7fa5941b6880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:22:36.310 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:22:36.307+0000 7fa5941b6880 -1 bluestore(td/test-erasure-code/1/block) _read_bdev_label failed to open td/test-erasure-code/1/block: (2) No such file or directory 2024-09-16T07:22:36.310 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:22:36.307+0000 7fa5941b6880 -1 bluestore(td/test-erasure-code/1/block) _read_bdev_label failed to open td/test-erasure-code/1/block: (2) No such file or directory 2024-09-16T07:22:36.310 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:22:36.307+0000 7fa5941b6880 -1 bluestore(td/test-erasure-code/1/block) _read_bdev_label failed to open td/test-erasure-code/1/block: (2) No such file or directory 2024-09-16T07:22:36.399 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:22:36.395+0000 7fa5941b6880 -1 bluestore(td/test-erasure-code/1) _read_fsid unparsable uuid 2024-09-16T07:22:42.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/1/keyring 2024-09-16T07:22:42.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:22:42.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T07:22:42.495 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-16T07:22:42.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:22:43.047 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-16T07:22:43.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T07:22:43.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:22:43.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:22:43.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:22:43.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:22:43.081 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:22:43.079+0000 7f74669b5880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:22:43.110 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:22:43.107+0000 7f74669b5880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:22:43.112 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:22:43.107+0000 7f74669b5880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:22:43.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T07:22:43.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:22:43.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T07:22:43.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:22:43.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:22:43.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:43.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:22:43.543 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:22:43.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:22:43.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:43.671 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:22:43.667+0000 7f74669b5880 -1 Falling back to public interface 2024-09-16T07:22:44.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:45.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:45.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:45.066 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:22:45.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:22:45.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:45.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:22:45.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:46.166 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:22:46.163+0000 7f74669b5880 -1 osd.1 0 log_to_monitors true 2024-09-16T07:22:46.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:46.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:46.485 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:22:46.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:22:46.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:46.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:22:47.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:48.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:48.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:48.067 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:22:48.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:22:48.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:48.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:22:48.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:49.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:49.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:49.579 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:22:49.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:22:49.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:49.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:22:50.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:51.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:51.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:51.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:22:51.112 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:22:51.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:51.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:22:51.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:52.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:52.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:52.538 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:22:52.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:22:52.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:52.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:22:53.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:54.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:54.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:54.070 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:22:54.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:22:54.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:54.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:22:54.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:55.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:55.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:55.589 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:22:55.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:22:55.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:55.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:22:56.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:57.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:57.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:57.082 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:22:57.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:22:57.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:57.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:22:57.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:22:58.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:22:58.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:22:58.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:22:58.544 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:22:58.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:22:58.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:22:59.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:00.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:00.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:00.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:23:00.030 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:23:00.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:00.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:23:00.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:01.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:01.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:01.550 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:23:01.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:23:01.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:01.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:23:02.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:03.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:03.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:03.515 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:23:03.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:23:03.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:03.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:23:04.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:05.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:05.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:05.188 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:23:05.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:23:05.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:05.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:23:06.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:07.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:07.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:07.041 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:23:07.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:23:07.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:07.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:23:08.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:09.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:09.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:09.175 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:23:09.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:23:09.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:09.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:23:09.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:10.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:10.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:10.778 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:23:10.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:23:10.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:10.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:23:11.237 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1103852135,v1:127.0.0.1:6811/1103852135] [v2:127.0.0.1:6812/1103852135,v1:127.0.0.1:6813/1103852135] exists,up 0811deaf-21fd-46c0-843f-bb43fa4bc866 2024-09-16T07:23:11.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:23:11.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:23:11.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:23:11.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-16T07:23:11.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 2 2024-09-16T07:23:11.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-16T07:23:11.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:23:11.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T07:23:11.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:23:11.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/2 2024-09-16T07:23:11.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-16T07:23:11.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:23:11.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:23:11.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:23:11.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/2' 2024-09-16T07:23:11.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/2/journal' 2024-09-16T07:23:11.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:23:11.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:23:11.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-16T07:23:11.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:23:11.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:23:11.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:23:11.245 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:23:11.245 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:23:11.245 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:23:11.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:23:11.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:23:11.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:23:11.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:23:11.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:23:11.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-16T07:23:11.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-16T07:23:11.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:23:11.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:23:11.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:23:11.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:23:11.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:23:11.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:23:11.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/2 2024-09-16T07:23:11.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:23:11.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0e07de27-e33c-45b6-b0db-ce8fab1c4b11 2024-09-16T07:23:11.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 0e07de27-e33c-45b6-b0db-ce8fab1c4b11' 2024-09-16T07:23:11.247 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 0e07de27-e33c-45b6-b0db-ce8fab1c4b11 2024-09-16T07:23:11.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:23:11.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDf3OdmLN8mEBAAuidT+Kj5uC7yXQa7smRVxQ== 2024-09-16T07:23:11.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDf3OdmLN8mEBAAuidT+Kj5uC7yXQa7smRVxQ=="}' 2024-09-16T07:23:11.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0e07de27-e33c-45b6-b0db-ce8fab1c4b11 -i td/test-erasure-code/2/new.json 2024-09-16T07:23:11.865 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:23:11.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/2/new.json 2024-09-16T07:23:11.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDf3OdmLN8mEBAAuidT+Kj5uC7yXQa7smRVxQ== --osd-uuid 0e07de27-e33c-45b6-b0db-ce8fab1c4b11 2024-09-16T07:23:11.921 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:23:11.922+0000 7fb6c9f13880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:23:11.945 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:23:11.946+0000 7fb6c9f13880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:23:11.947 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:23:11.946+0000 7fb6c9f13880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:23:11.948 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:23:11.950+0000 7fb6c9f13880 -1 bluestore(td/test-erasure-code/2/block) _read_bdev_label failed to open td/test-erasure-code/2/block: (2) No such file or directory 2024-09-16T07:23:11.948 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:23:11.950+0000 7fb6c9f13880 -1 bluestore(td/test-erasure-code/2/block) _read_bdev_label failed to open td/test-erasure-code/2/block: (2) No such file or directory 2024-09-16T07:23:11.949 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:23:11.950+0000 7fb6c9f13880 -1 bluestore(td/test-erasure-code/2/block) _read_bdev_label failed to open td/test-erasure-code/2/block: (2) No such file or directory 2024-09-16T07:23:12.033 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:23:12.034+0000 7fb6c9f13880 -1 bluestore(td/test-erasure-code/2) _read_fsid unparsable uuid 2024-09-16T07:23:18.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/2/keyring 2024-09-16T07:23:18.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:23:18.211 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-16T07:23:18.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T07:23:18.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:23:18.771 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-16T07:23:18.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T07:23:18.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:23:18.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:23:18.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:23:18.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:23:18.805 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:23:18.806+0000 7f9307b43880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:23:18.827 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:23:18.826+0000 7f9307b43880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:23:18.828 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:23:18.830+0000 7f9307b43880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:23:19.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T07:23:19.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:23:19.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T07:23:19.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:23:19.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:23:19.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:19.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:23:19.200 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:23:19.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:19.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:23:19.395 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:23:19.394+0000 7f9307b43880 -1 Falling back to public interface 2024-09-16T07:23:19.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:20.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:20.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:20.664 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:23:20.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:23:20.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:20.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:23:21.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:21.876 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:23:21.874+0000 7f9307b43880 -1 osd.2 0 log_to_monitors true 2024-09-16T07:23:22.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:22.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:22.089 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:23:22.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:23:22.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:22.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:23:22.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:23.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:23.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:23.596 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:23:23.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:23:23.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:23.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:23:24.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:25.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:25.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:25.026 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:23:25.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:23:25.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:25.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:23:25.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:26.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:26.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:26.450 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:23:26.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:23:26.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:26.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:23:26.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:27.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:27.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:27.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:23:27.997 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:23:27.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:27.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:23:28.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:29.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:29.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:29.485 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:23:29.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:23:29.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:29.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:23:30.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:31.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:31.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:31.098 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:23:31.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:23:31.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:31.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:23:31.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:32.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:32.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:32.684 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:23:32.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:23:32.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:32.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:23:33.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:34.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:34.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:34.088 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:23:34.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:23:34.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:34.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:23:35.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:36.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:36.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:36.087 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:23:36.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:23:36.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:36.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:23:36.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:37.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:37.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:37.715 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:23:37.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:23:37.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:37.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:23:38.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:39.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:39.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:39.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:23:39.612 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:23:39.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:39.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:23:40.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:41.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:41.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:41.529 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:23:41.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:23:41.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:41.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:23:42.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:43.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:43.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:43.542 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:23:43.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:23:43.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:43.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:23:44.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:45.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:45.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:45.422 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:23:45.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:23:45.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:45.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:23:45.924 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3301363534,v1:127.0.0.1:6819/3301363534] [v2:127.0.0.1:6820/3301363534,v1:127.0.0.1:6821/3301363534] exists,up 0e07de27-e33c-45b6-b0db-ce8fab1c4b11 2024-09-16T07:23:45.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:23:45.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:23:45.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:23:45.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-16T07:23:45.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 3 2024-09-16T07:23:45.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-16T07:23:45.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:23:45.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T07:23:45.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:23:45.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/3 2024-09-16T07:23:45.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-16T07:23:45.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:23:45.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:23:45.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:23:45.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2024-09-16T07:23:45.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2024-09-16T07:23:45.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:23:45.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:23:45.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-16T07:23:45.928 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:23:45.928 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:23:45.928 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:23:45.928 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:23:45.928 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:23:45.928 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:23:45.930 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:23:45.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:23:45.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:23:45.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:23:45.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:23:45.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-16T07:23:45.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-16T07:23:45.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:23:45.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:23:45.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:23:45.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:23:45.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:23:45.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:23:45.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/3 2024-09-16T07:23:45.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:23:45.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2126ebea-dac7-4748-8e6b-84428d3991be 2024-09-16T07:23:45.933 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 2126ebea-dac7-4748-8e6b-84428d3991be 2024-09-16T07:23:45.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 2126ebea-dac7-4748-8e6b-84428d3991be' 2024-09-16T07:23:45.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:23:45.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAB3edmprY+ORAACMINDIju6OoW08q7In+d7A== 2024-09-16T07:23:45.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAB3edmprY+ORAACMINDIju6OoW08q7In+d7A=="}' 2024-09-16T07:23:45.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2126ebea-dac7-4748-8e6b-84428d3991be -i td/test-erasure-code/3/new.json 2024-09-16T07:23:46.569 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:23:46.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/3/new.json 2024-09-16T07:23:46.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAB3edmprY+ORAACMINDIju6OoW08q7In+d7A== --osd-uuid 2126ebea-dac7-4748-8e6b-84428d3991be 2024-09-16T07:23:46.624 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:23:46.621+0000 7f7b75731880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:23:46.692 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:23:46.689+0000 7f7b75731880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:23:46.694 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:23:46.693+0000 7f7b75731880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:23:46.695 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:23:46.693+0000 7f7b75731880 -1 bluestore(td/test-erasure-code/3/block) _read_bdev_label failed to open td/test-erasure-code/3/block: (2) No such file or directory 2024-09-16T07:23:46.696 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:23:46.693+0000 7f7b75731880 -1 bluestore(td/test-erasure-code/3/block) _read_bdev_label failed to open td/test-erasure-code/3/block: (2) No such file or directory 2024-09-16T07:23:46.696 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:23:46.693+0000 7f7b75731880 -1 bluestore(td/test-erasure-code/3/block) _read_bdev_label failed to open td/test-erasure-code/3/block: (2) No such file or directory 2024-09-16T07:23:46.783 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:23:46.781+0000 7f7b75731880 -1 bluestore(td/test-erasure-code/3) _read_fsid unparsable uuid 2024-09-16T07:23:53.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/3/keyring 2024-09-16T07:23:53.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:23:53.019 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-16T07:23:53.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T07:23:53.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:23:53.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T07:23:53.579 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-16T07:23:53.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:23:53.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:23:53.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:23:53.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:23:53.611 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:23:53.609+0000 7f5b083e6880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:23:53.614 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:23:53.613+0000 7f5b083e6880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:23:53.616 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:23:53.613+0000 7f5b083e6880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:23:53.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T07:23:53.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:23:53.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T07:23:53.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:23:53.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:23:53.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:53.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:23:53.992 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:23:53.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:53.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:23:54.187 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:23:54.185+0000 7f5b083e6880 -1 Falling back to public interface 2024-09-16T07:23:54.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:55.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:55.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:55.529 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:23:55.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:23:55.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:55.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:23:55.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:56.774 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:23:56.773+0000 7f5b083e6880 -1 osd.3 0 log_to_monitors true 2024-09-16T07:23:56.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:56.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:56.979 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:23:56.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:23:56.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:56.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:23:57.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:23:58.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:23:58.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:23:58.546 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:23:58.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:23:58.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:23:58.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:23:59.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:00.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:00.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:00.117 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:24:00.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:24:00.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:00.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:24:00.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:01.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:01.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:01.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:24:01.588 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:24:01.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:01.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:24:02.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:03.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:03.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:03.032 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:24:03.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:24:03.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:03.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:24:03.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:04.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:04.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:04.454 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:24:04.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:24:04.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:04.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:24:04.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:05.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:05.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:05.959 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:24:05.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:24:05.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:05.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:24:06.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:07.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:07.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:07.442 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:24:07.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:24:07.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:07.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:24:07.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:08.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:08.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:08.977 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:24:08.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:24:08.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:08.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:24:09.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:10.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:10.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:10.412 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:24:10.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:24:10.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:10.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:24:11.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:12.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:12.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:12.205 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:24:12.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:24:12.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:12.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:24:12.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:13.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:13.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:13.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:24:13.769 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:24:13.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:13.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:24:14.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:15.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:15.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:15.767 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:24:15.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:24:15.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:15.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:24:16.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:17.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:17.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:17.631 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:24:17.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:24:17.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:17.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:24:18.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:19.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:19.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:19.536 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:24:19.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:24:19.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:24:20.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:21.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:21.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:21.303 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:24:21.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:24:21.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:21.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:24:21.783 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1535083776,v1:127.0.0.1:6827/1535083776] [v2:127.0.0.1:6828/1535083776,v1:127.0.0.1:6829/1535083776] exists,up 2126ebea-dac7-4748-8e6b-84428d3991be 2024-09-16T07:24:21.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:24:21.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:24:21.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:24:21.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-16T07:24:21.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 4 2024-09-16T07:24:21.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-16T07:24:21.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:24:21.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-16T07:24:21.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:24:21.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/4 2024-09-16T07:24:21.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-16T07:24:21.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:24:21.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:24:21.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:24:21.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/4' 2024-09-16T07:24:21.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/4/journal' 2024-09-16T07:24:21.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:24:21.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:24:21.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-16T07:24:21.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:24:21.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:24:21.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:24:21.787 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:24:21.787 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:24:21.787 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:24:21.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:24:21.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:24:21.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:24:21.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:24:21.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:24:21.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-16T07:24:21.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-16T07:24:21.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:24:21.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:24:21.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:24:21.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:24:21.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:24:21.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:24:21.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/4 2024-09-16T07:24:21.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:24:21.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7768d9f2-a52c-428c-b473-f448e4942280 2024-09-16T07:24:21.790 INFO:tasks.workunit.client.0.smithi162.stdout:add osd4 7768d9f2-a52c-428c-b473-f448e4942280 2024-09-16T07:24:21.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 7768d9f2-a52c-428c-b473-f448e4942280' 2024-09-16T07:24:21.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:24:21.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAl3edmwSifMBAAWICKPkQIHNSu5wtG/yL/7w== 2024-09-16T07:24:21.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAl3edmwSifMBAAWICKPkQIHNSu5wtG/yL/7w=="}' 2024-09-16T07:24:21.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7768d9f2-a52c-428c-b473-f448e4942280 -i td/test-erasure-code/4/new.json 2024-09-16T07:24:22.434 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:24:22.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/4/new.json 2024-09-16T07:24:22.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAl3edmwSifMBAAWICKPkQIHNSu5wtG/yL/7w== --osd-uuid 7768d9f2-a52c-428c-b473-f448e4942280 2024-09-16T07:24:22.482 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:24:22.480+0000 7f90b01ba880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:24:22.510 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:24:22.504+0000 7f90b01ba880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:24:22.513 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:24:22.512+0000 7f90b01ba880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:24:22.513 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:24:22.512+0000 7f90b01ba880 -1 bluestore(td/test-erasure-code/4/block) _read_bdev_label failed to open td/test-erasure-code/4/block: (2) No such file or directory 2024-09-16T07:24:22.514 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:24:22.512+0000 7f90b01ba880 -1 bluestore(td/test-erasure-code/4/block) _read_bdev_label failed to open td/test-erasure-code/4/block: (2) No such file or directory 2024-09-16T07:24:22.514 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:24:22.512+0000 7f90b01ba880 -1 bluestore(td/test-erasure-code/4/block) _read_bdev_label failed to open td/test-erasure-code/4/block: (2) No such file or directory 2024-09-16T07:24:22.639 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:24:22.640+0000 7f90b01ba880 -1 bluestore(td/test-erasure-code/4) _read_fsid unparsable uuid 2024-09-16T07:24:28.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/4/keyring 2024-09-16T07:24:28.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:24:28.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-16T07:24:28.736 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd4 key to auth repository 2024-09-16T07:24:28.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:24:29.302 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.4 2024-09-16T07:24:29.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-16T07:24:29.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:24:29.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:24:29.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:24:29.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:24:29.335 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:24:29.332+0000 7f9f3f487880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:24:29.338 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:24:29.336+0000 7f9f3f487880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:24:29.340 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:24:29.336+0000 7f9f3f487880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:24:29.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-16T07:24:29.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:24:29.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-16T07:24:29.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:24:29.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:24:29.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:29.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:24:29.789 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:24:29.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:29.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:24:29.886 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:24:29.884+0000 7f9f3f487880 -1 Falling back to public interface 2024-09-16T07:24:30.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:31.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:31.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:31.242 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:24:31.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:24:31.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:31.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:24:31.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:32.489 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:24:32.488+0000 7f9f3f487880 -1 osd.4 0 log_to_monitors true 2024-09-16T07:24:32.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:32.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:32.666 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:24:32.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:24:32.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:32.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:24:33.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:34.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:34.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:34.144 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:24:34.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:24:34.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:34.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:24:34.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:35.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:35.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:35.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:24:35.721 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:24:35.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:35.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:24:36.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:37.275 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:24:37.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:37.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:37.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:24:37.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:37.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:24:37.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:38.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:38.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:38.833 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:24:38.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:24:38.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:38.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:24:39.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:40.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:40.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:40.324 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:24:40.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:24:40.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:40.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:24:40.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:41.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:41.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:41.914 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:24:41.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:24:41.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:41.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:24:42.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:43.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:43.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:43.379 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:24:43.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:24:43.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:43.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:24:43.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:44.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:44.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:44.818 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:24:44.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:24:44.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:44.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:24:45.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:46.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:46.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:46.439 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:24:46.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:24:46.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:46.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:24:47.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:48.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:48.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:48.574 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:24:48.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:24:48.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:48.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:24:49.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:50.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:50.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:50.149 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:24:50.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:24:50.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:50.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:24:50.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:51.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:51.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:51.666 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:24:51.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:24:51.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:51.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:24:52.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:53.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:53.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:53.190 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:24:53.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:24:53.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:53.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:24:54.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:55.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:55.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:55.275 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:24:55.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:24:55.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:55.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:24:55.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:24:56.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:24:56.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:24:56.844 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:24:56.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:24:56.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:24:56.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:24:57.299 INFO:tasks.workunit.client.0.smithi162.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1331838149,v1:127.0.0.1:6835/1331838149] [v2:127.0.0.1:6836/1331838149,v1:127.0.0.1:6837/1331838149] exists,up 7768d9f2-a52c-428c-b473-f448e4942280 2024-09-16T07:24:57.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:24:57.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:24:57.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:24:57.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-16T07:24:57.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 5 2024-09-16T07:24:57.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-16T07:24:57.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:24:57.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-16T07:24:57.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:24:57.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/5 2024-09-16T07:24:57.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-16T07:24:57.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:24:57.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:24:57.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:24:57.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2024-09-16T07:24:57.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2024-09-16T07:24:57.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:24:57.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:24:57.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-16T07:24:57.304 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:24:57.304 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:24:57.304 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:24:57.304 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:24:57.304 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:24:57.304 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:24:57.304 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:24:57.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:24:57.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:24:57.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:24:57.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:24:57.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-16T07:24:57.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-16T07:24:57.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:24:57.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:24:57.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:24:57.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:24:57.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:24:57.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:24:57.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/5 2024-09-16T07:24:57.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:24:57.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e359a44a-e8ed-412a-942a-e6133f52f94c 2024-09-16T07:24:57.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 e359a44a-e8ed-412a-942a-e6133f52f94c' 2024-09-16T07:24:57.307 INFO:tasks.workunit.client.0.smithi162.stdout:add osd5 e359a44a-e8ed-412a-942a-e6133f52f94c 2024-09-16T07:24:57.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:24:57.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBJ3edmjweOExAA209B2EdT9Pp1g0FBbwV37Q== 2024-09-16T07:24:57.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBJ3edmjweOExAA209B2EdT9Pp1g0FBbwV37Q=="}' 2024-09-16T07:24:57.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e359a44a-e8ed-412a-942a-e6133f52f94c -i td/test-erasure-code/5/new.json 2024-09-16T07:24:57.883 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:24:57.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/5/new.json 2024-09-16T07:24:57.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBJ3edmjweOExAA209B2EdT9Pp1g0FBbwV37Q== --osd-uuid e359a44a-e8ed-412a-942a-e6133f52f94c 2024-09-16T07:24:57.929 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:24:57.928+0000 7fa5373c9880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:24:57.933 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:24:57.932+0000 7fa5373c9880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:24:57.935 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:24:57.932+0000 7fa5373c9880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:24:57.936 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:24:57.932+0000 7fa5373c9880 -1 bluestore(td/test-erasure-code/5/block) _read_bdev_label failed to open td/test-erasure-code/5/block: (2) No such file or directory 2024-09-16T07:24:57.936 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:24:57.932+0000 7fa5373c9880 -1 bluestore(td/test-erasure-code/5/block) _read_bdev_label failed to open td/test-erasure-code/5/block: (2) No such file or directory 2024-09-16T07:24:57.936 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:24:57.932+0000 7fa5373c9880 -1 bluestore(td/test-erasure-code/5/block) _read_bdev_label failed to open td/test-erasure-code/5/block: (2) No such file or directory 2024-09-16T07:24:58.026 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:24:58.024+0000 7fa5373c9880 -1 bluestore(td/test-erasure-code/5) _read_fsid unparsable uuid 2024-09-16T07:25:04.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/5/keyring 2024-09-16T07:25:04.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:25:04.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-16T07:25:04.139 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd5 key to auth repository 2024-09-16T07:25:04.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:25:04.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-16T07:25:04.726 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.5 2024-09-16T07:25:04.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:25:04.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:25:04.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:25:04.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:25:04.752 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:25:04.748+0000 7f85bc24b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:25:04.776 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:25:04.772+0000 7f85bc24b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:25:04.778 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:25:04.776+0000 7f85bc24b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:25:05.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-16T07:25:05.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:25:05.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-16T07:25:05.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:25:05.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:25:05.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:05.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:25:05.165 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:25:05.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:25:05.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:05.355 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:25:05.351+0000 7f85bc24b880 -1 Falling back to public interface 2024-09-16T07:25:05.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:06.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:06.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:06.627 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:25:06.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:25:06.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:06.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:25:07.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:08.021 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:25:08.019+0000 7f85bc24b880 -1 osd.5 0 log_to_monitors true 2024-09-16T07:25:08.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:08.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:08.058 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:25:08.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:25:08.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:08.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:25:08.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:09.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:09.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:09.777 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:25:09.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:25:09.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:09.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:25:10.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:11.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:11.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:11.200 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:25:11.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:25:11.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:11.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:25:11.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:12.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:12.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:12.637 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:25:12.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:25:12.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:12.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:25:13.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:14.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:14.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:14.223 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:25:14.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:25:14.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:14.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:25:14.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:15.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:15.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:15.820 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:25:15.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:25:15.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:15.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:25:16.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:17.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:17.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:17.298 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:25:17.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:25:17.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:17.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:25:17.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:18.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:18.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:18.983 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:25:18.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:25:18.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:18.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:25:19.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:20.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:20.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:20.433 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:25:20.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:25:20.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:20.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:25:21.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:22.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:22.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:22.339 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:25:22.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:25:22.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:22.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:25:22.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:23.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:23.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:23.931 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:25:23.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:25:23.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:23.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:25:24.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:25.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:25.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:25.745 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:25:25.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:25:25.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:25.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:25:26.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:27.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:27.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:27.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:25:27.506 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:25:27.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:27.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:25:28.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:29.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:29.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:29.412 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:25:29.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:25:29.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:29.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:25:30.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:31.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:31.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:31.256 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:25:31.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:25:31.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:31.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:25:31.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:32.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:32.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:32.929 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:25:32.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:25:32.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:32.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:25:33.479 INFO:tasks.workunit.client.0.smithi162.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2471206994,v1:127.0.0.1:6843/2471206994] [v2:127.0.0.1:6844/2471206994,v1:127.0.0.1:6845/2471206994] exists,up e359a44a-e8ed-412a-942a-e6133f52f94c 2024-09-16T07:25:33.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:25:33.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:25:33.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:25:33.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-16T07:25:33.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 6 2024-09-16T07:25:33.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-16T07:25:33.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:25:33.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-09-16T07:25:33.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:25:33.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/6 2024-09-16T07:25:33.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-16T07:25:33.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:25:33.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:25:33.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:25:33.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2024-09-16T07:25:33.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2024-09-16T07:25:33.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:25:33.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:25:33.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-16T07:25:33.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:25:33.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:25:33.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:25:33.484 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:25:33.485 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:25:33.485 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:25:33.485 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:25:33.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:25:33.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:25:33.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:25:33.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:25:33.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-16T07:25:33.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-16T07:25:33.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:25:33.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:25:33.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:25:33.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:25:33.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:25:33.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:25:33.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/6 2024-09-16T07:25:33.486 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:25:33.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=377c6e19-55cf-4eee-a1da-650f297ae4c6 2024-09-16T07:25:33.487 INFO:tasks.workunit.client.0.smithi162.stdout:add osd6 377c6e19-55cf-4eee-a1da-650f297ae4c6 2024-09-16T07:25:33.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 377c6e19-55cf-4eee-a1da-650f297ae4c6' 2024-09-16T07:25:33.488 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:25:33.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBt3edmBW9mHhAADw3mZJ2AO+T/82oY3kS79Q== 2024-09-16T07:25:33.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBt3edmBW9mHhAADw3mZJ2AO+T/82oY3kS79Q=="}' 2024-09-16T07:25:33.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 377c6e19-55cf-4eee-a1da-650f297ae4c6 -i td/test-erasure-code/6/new.json 2024-09-16T07:25:34.108 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:25:34.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/6/new.json 2024-09-16T07:25:34.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBt3edmBW9mHhAADw3mZJ2AO+T/82oY3kS79Q== --osd-uuid 377c6e19-55cf-4eee-a1da-650f297ae4c6 2024-09-16T07:25:34.163 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:25:34.159+0000 7febb7e3d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:25:34.167 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:25:34.163+0000 7febb7e3d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:25:34.169 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:25:34.167+0000 7febb7e3d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:25:34.170 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:25:34.167+0000 7febb7e3d880 -1 bluestore(td/test-erasure-code/6/block) _read_bdev_label failed to open td/test-erasure-code/6/block: (2) No such file or directory 2024-09-16T07:25:34.170 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:25:34.167+0000 7febb7e3d880 -1 bluestore(td/test-erasure-code/6/block) _read_bdev_label failed to open td/test-erasure-code/6/block: (2) No such file or directory 2024-09-16T07:25:34.170 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:25:34.167+0000 7febb7e3d880 -1 bluestore(td/test-erasure-code/6/block) _read_bdev_label failed to open td/test-erasure-code/6/block: (2) No such file or directory 2024-09-16T07:25:34.256 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:25:34.251+0000 7febb7e3d880 -1 bluestore(td/test-erasure-code/6) _read_fsid unparsable uuid 2024-09-16T07:25:40.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/6/keyring 2024-09-16T07:25:40.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:25:40.499 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd6 key to auth repository 2024-09-16T07:25:40.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-09-16T07:25:40.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:25:41.060 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.6 2024-09-16T07:25:41.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-09-16T07:25:41.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:25:41.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:25:41.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:25:41.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:25:41.094 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:25:41.091+0000 7f9e1338a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:25:41.123 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:25:41.119+0000 7f9e1338a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:25:41.125 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:25:41.123+0000 7f9e1338a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:25:41.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-09-16T07:25:41.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:25:41.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-09-16T07:25:41.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:25:41.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:25:41.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:41.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:25:41.502 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:25:41.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:41.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:25:41.683 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:25:41.679+0000 7f9e1338a880 -1 Falling back to public interface 2024-09-16T07:25:41.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:42.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:42.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:42.996 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:25:42.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:25:42.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:42.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:25:43.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:44.293 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:25:44.291+0000 7f9e1338a880 -1 osd.6 0 log_to_monitors true 2024-09-16T07:25:44.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:44.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:44.465 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:25:44.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:25:44.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:44.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:25:44.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:45.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:45.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:45.984 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:25:45.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:25:45.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:45.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:25:46.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:47.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:47.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:47.632 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:25:47.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:25:47.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:47.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:25:48.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:49.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:49.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:49.237 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:25:49.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:25:49.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:49.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:25:49.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:50.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:50.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:50.716 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:25:50.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:25:50.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:50.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:25:51.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:52.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:52.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:52.142 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:25:52.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:25:52.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:52.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:25:52.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:53.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:53.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:53.673 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:25:53.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:25:53.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:53.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:25:54.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:55.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:55.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:55.253 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:25:55.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:25:55.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:55.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:25:55.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:56.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:56.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:56.682 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:25:56.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:25:56.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:56.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:25:57.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:58.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:58.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:58.284 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:25:58.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:25:58.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:58.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:25:58.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:25:59.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:25:59.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:25:59.818 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:25:59.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:25:59.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:25:59.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:26:00.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:01.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:01.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:01.949 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:26:01.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:26:01.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:01.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:26:02.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:03.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:03.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:03.690 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:26:03.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:26:03.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:03.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:26:04.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:05.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:05.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:05.375 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:26:05.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:26:05.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:05.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:26:06.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:07.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:07.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:07.193 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:26:07.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:26:07.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:07.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:26:07.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:08.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:08.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:08.949 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:26:08.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:26:08.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:08.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:26:09.402 INFO:tasks.workunit.client.0.smithi162.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/166806526,v1:127.0.0.1:6851/166806526] [v2:127.0.0.1:6852/166806526,v1:127.0.0.1:6853/166806526] exists,up 377c6e19-55cf-4eee-a1da-650f297ae4c6 2024-09-16T07:26:09.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:26:09.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:26:09.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:26:09.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-16T07:26:09.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 7 2024-09-16T07:26:09.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-16T07:26:09.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:26:09.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-09-16T07:26:09.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:26:09.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/7 2024-09-16T07:26:09.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-16T07:26:09.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:26:09.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:26:09.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:26:09.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/7' 2024-09-16T07:26:09.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/7/journal' 2024-09-16T07:26:09.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:26:09.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:26:09.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-16T07:26:09.407 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:26:09.407 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:26:09.407 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:26:09.407 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:26:09.407 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:26:09.407 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:26:09.407 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:26:09.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:26:09.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:26:09.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:26:09.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:26:09.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-16T07:26:09.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-16T07:26:09.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:26:09.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:26:09.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:26:09.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:26:09.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:26:09.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:26:09.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/7 2024-09-16T07:26:09.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:26:09.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=61f5b42c-63a2-4b27-bde7-66202bade983 2024-09-16T07:26:09.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 61f5b42c-63a2-4b27-bde7-66202bade983' 2024-09-16T07:26:09.413 INFO:tasks.workunit.client.0.smithi162.stdout:add osd7 61f5b42c-63a2-4b27-bde7-66202bade983 2024-09-16T07:26:09.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:26:09.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCR3edmSFHWGRAALDcfqCZD0gqZjrOpmeMHEQ== 2024-09-16T07:26:09.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCR3edmSFHWGRAALDcfqCZD0gqZjrOpmeMHEQ=="}' 2024-09-16T07:26:09.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 61f5b42c-63a2-4b27-bde7-66202bade983 -i td/test-erasure-code/7/new.json 2024-09-16T07:26:10.172 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:26:10.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/7/new.json 2024-09-16T07:26:10.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCR3edmSFHWGRAALDcfqCZD0gqZjrOpmeMHEQ== --osd-uuid 61f5b42c-63a2-4b27-bde7-66202bade983 2024-09-16T07:26:10.226 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:26:10.226+0000 7fac98db9880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:26:10.237 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:26:10.238+0000 7fac98db9880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:26:10.239 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:26:10.238+0000 7fac98db9880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:26:10.240 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:26:10.238+0000 7fac98db9880 -1 bluestore(td/test-erasure-code/7/block) _read_bdev_label failed to open td/test-erasure-code/7/block: (2) No such file or directory 2024-09-16T07:26:10.240 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:26:10.238+0000 7fac98db9880 -1 bluestore(td/test-erasure-code/7/block) _read_bdev_label failed to open td/test-erasure-code/7/block: (2) No such file or directory 2024-09-16T07:26:10.240 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:26:10.238+0000 7fac98db9880 -1 bluestore(td/test-erasure-code/7/block) _read_bdev_label failed to open td/test-erasure-code/7/block: (2) No such file or directory 2024-09-16T07:26:10.411 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:26:10.410+0000 7fac98db9880 -1 bluestore(td/test-erasure-code/7) _read_fsid unparsable uuid 2024-09-16T07:26:16.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/7/keyring 2024-09-16T07:26:16.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:26:16.867 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd7 key to auth repository 2024-09-16T07:26:16.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-09-16T07:26:16.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:26:17.432 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.7 2024-09-16T07:26:17.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-09-16T07:26:17.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:26:17.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:26:17.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:26:17.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:26:17.468 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:26:17.466+0000 7fa49ed9a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:26:17.491 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:26:17.490+0000 7fa49ed9a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:26:17.493 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:26:17.494+0000 7fa49ed9a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:26:17.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-09-16T07:26:17.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:26:17.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-09-16T07:26:17.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:26:17.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:26:17.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:17.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:26:17.931 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:26:17.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:17.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:26:18.062 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:26:18.062+0000 7fa49ed9a880 -1 Falling back to public interface 2024-09-16T07:26:18.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:19.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:19.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:19.353 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:26:19.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:26:19.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:19.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:26:19.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:20.617 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:26:20.618+0000 7fa49ed9a880 -1 osd.7 0 log_to_monitors true 2024-09-16T07:26:20.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:20.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:20.811 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:26:20.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:26:20.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:20.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:26:21.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:22.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:22.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:22.339 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:26:22.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:26:22.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:22.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:26:22.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:23.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:23.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:23.852 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:26:23.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:26:23.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:23.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:26:24.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:25.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:25.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:25.430 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:26:25.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:26:25.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:25.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:26:25.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:26.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:26.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:26.929 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:26:26.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:26:26.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:26.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:26:27.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:28.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:28.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:28.460 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:26:28.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:26:28.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:28.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:26:28.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:29.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:29.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:29.917 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:26:29.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:26:29.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:29.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:26:30.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:31.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:31.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:31.447 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:26:31.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:26:31.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:31.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:26:31.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:32.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:32.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:32.860 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:26:32.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:26:32.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:32.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:26:33.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:34.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:34.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:34.558 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:26:34.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:26:34.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:34.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:26:35.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:36.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:36.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:36.097 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:26:36.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:26:36.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:36.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:26:36.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:37.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:37.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:37.930 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:26:37.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:26:37.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:37.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:26:38.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:39.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:39.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:39.402 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:26:39.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:26:39.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:39.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:26:40.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:41.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:41.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:41.481 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:26:41.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:26:41.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:41.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:26:42.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:43.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:43.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:43.193 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:26:43.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:26:43.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:43.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:26:43.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:44.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:44.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:44.806 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:26:44.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:26:44.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:44.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:26:45.255 INFO:tasks.workunit.client.0.smithi162.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/431150696,v1:127.0.0.1:6859/431150696] [v2:127.0.0.1:6860/431150696,v1:127.0.0.1:6861/431150696] exists,up 61f5b42c-63a2-4b27-bde7-66202bade983 2024-09-16T07:26:45.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:26:45.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:26:45.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:26:45.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-16T07:26:45.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 8 2024-09-16T07:26:45.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-16T07:26:45.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:26:45.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-09-16T07:26:45.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:26:45.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/8 2024-09-16T07:26:45.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-16T07:26:45.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:26:45.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:26:45.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:26:45.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/8' 2024-09-16T07:26:45.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/8/journal' 2024-09-16T07:26:45.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:26:45.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:26:45.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-16T07:26:45.261 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:26:45.261 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:26:45.261 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:26:45.261 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:26:45.261 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:26:45.261 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:26:45.261 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:26:45.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:26:45.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:26:45.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:26:45.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:26:45.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-16T07:26:45.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-16T07:26:45.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:26:45.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:26:45.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:26:45.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:26:45.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:26:45.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:26:45.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/8 2024-09-16T07:26:45.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:26:45.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6cc55787-bc7b-4756-a5ad-f7c6711a4e79 2024-09-16T07:26:45.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 6cc55787-bc7b-4756-a5ad-f7c6711a4e79' 2024-09-16T07:26:45.264 INFO:tasks.workunit.client.0.smithi162.stdout:add osd8 6cc55787-bc7b-4756-a5ad-f7c6711a4e79 2024-09-16T07:26:45.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:26:45.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC13edmaZUNERAAlPZ12jyx8HU0loJnPRSPCg== 2024-09-16T07:26:45.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC13edmaZUNERAAlPZ12jyx8HU0loJnPRSPCg=="}' 2024-09-16T07:26:45.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6cc55787-bc7b-4756-a5ad-f7c6711a4e79 -i td/test-erasure-code/8/new.json 2024-09-16T07:26:45.825 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:26:45.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/8/new.json 2024-09-16T07:26:45.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC13edmaZUNERAAlPZ12jyx8HU0loJnPRSPCg== --osd-uuid 6cc55787-bc7b-4756-a5ad-f7c6711a4e79 2024-09-16T07:26:45.877 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:26:45.873+0000 7ff7005cf880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:26:45.927 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:26:45.925+0000 7ff7005cf880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:26:45.929 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:26:45.929+0000 7ff7005cf880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:26:45.930 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:26:45.929+0000 7ff7005cf880 -1 bluestore(td/test-erasure-code/8/block) _read_bdev_label failed to open td/test-erasure-code/8/block: (2) No such file or directory 2024-09-16T07:26:45.930 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:26:45.929+0000 7ff7005cf880 -1 bluestore(td/test-erasure-code/8/block) _read_bdev_label failed to open td/test-erasure-code/8/block: (2) No such file or directory 2024-09-16T07:26:45.930 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:26:45.929+0000 7ff7005cf880 -1 bluestore(td/test-erasure-code/8/block) _read_bdev_label failed to open td/test-erasure-code/8/block: (2) No such file or directory 2024-09-16T07:26:45.985 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:26:45.985+0000 7ff7005cf880 -1 bluestore(td/test-erasure-code/8) _read_fsid unparsable uuid 2024-09-16T07:26:52.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/8/keyring 2024-09-16T07:26:52.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:26:52.120 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd8 key to auth repository 2024-09-16T07:26:52.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-09-16T07:26:52.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:26:52.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-09-16T07:26:52.701 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.8 2024-09-16T07:26:52.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:26:52.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:26:52.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:26:52.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:26:52.727 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:26:52.725+0000 7efc8b970880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:26:52.729 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:26:52.729+0000 7efc8b970880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:26:52.732 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:26:52.729+0000 7efc8b970880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:26:53.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-09-16T07:26:53.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:26:53.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-09-16T07:26:53.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:26:53.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:26:53.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:53.167 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:26:53.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:26:53.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:53.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:26:53.280 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:26:53.277+0000 7efc8b970880 -1 Falling back to public interface 2024-09-16T07:26:53.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:54.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:54.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:54.582 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:26:54.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:26:54.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:54.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:26:55.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:55.915 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:26:55.913+0000 7efc8b970880 -1 osd.8 0 log_to_monitors true 2024-09-16T07:26:56.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:56.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:56.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:26:56.079 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:26:56.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:56.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:26:56.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:57.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:57.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:57.628 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:26:57.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:26:57.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:57.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:26:58.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:26:59.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:26:59.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:26:59.082 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:26:59.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:26:59.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:26:59.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:26:59.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:00.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:00.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:00.510 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:27:00.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:27:00.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:00.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:27:00.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:01.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:01.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:01.956 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:27:01.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:27:01.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:01.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:27:02.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:03.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:03.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:03.398 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:27:03.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:27:03.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:03.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:27:03.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:04.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:04.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:04.991 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:27:04.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:27:04.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:04.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:27:05.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:06.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:06.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:06.424 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:27:06.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:27:06.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:06.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:27:06.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:07.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:07.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:07.950 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:27:07.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:27:07.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:27:07.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:08.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:09.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:09.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:09.506 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:27:09.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:27:09.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:09.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:27:10.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:11.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:11.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:11.371 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:27:11.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:27:11.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:11.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:27:12.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:13.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:13.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:13.431 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:27:13.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:27:13.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:13.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:27:14.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:15.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:15.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:15.650 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:27:15.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:27:15.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:15.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:27:16.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:17.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:17.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:17.530 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:27:17.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:27:17.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:17.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:27:18.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:19.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:19.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:19.582 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:27:19.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:27:19.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:19.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:27:20.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:21.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:21.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:21.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:27:21.057 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:27:21.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:21.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:27:21.484 INFO:tasks.workunit.client.0.smithi162.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/713094960,v1:127.0.0.1:6867/713094960] [v2:127.0.0.1:6868/713094960,v1:127.0.0.1:6869/713094960] exists,up 6cc55787-bc7b-4756-a5ad-f7c6711a4e79 2024-09-16T07:27:21.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:27:21.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:27:21.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:27:21.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-16T07:27:21.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 9 2024-09-16T07:27:21.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-16T07:27:21.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:27:21.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-09-16T07:27:21.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:27:21.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/9 2024-09-16T07:27:21.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-16T07:27:21.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:27:21.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:27:21.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:27:21.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2024-09-16T07:27:21.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2024-09-16T07:27:21.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:27:21.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:27:21.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-16T07:27:21.488 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:27:21.489 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:27:21.489 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:27:21.490 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:27:21.490 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:27:21.490 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:27:21.490 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:27:21.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:27:21.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:27:21.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:27:21.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:27:21.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-16T07:27:21.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-16T07:27:21.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:27:21.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:27:21.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:27:21.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:27:21.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:27:21.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:27:21.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/9 2024-09-16T07:27:21.492 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:27:21.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1219e850-bad2-4898-8efe-09fa3f2b1315 2024-09-16T07:27:21.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 1219e850-bad2-4898-8efe-09fa3f2b1315' 2024-09-16T07:27:21.492 INFO:tasks.workunit.client.0.smithi162.stdout:add osd9 1219e850-bad2-4898-8efe-09fa3f2b1315 2024-09-16T07:27:21.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:27:21.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDZ3edmle3aHhAACvqAx2vb5FhP4LoamJQhMA== 2024-09-16T07:27:21.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDZ3edmle3aHhAACvqAx2vb5FhP4LoamJQhMA=="}' 2024-09-16T07:27:21.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1219e850-bad2-4898-8efe-09fa3f2b1315 -i td/test-erasure-code/9/new.json 2024-09-16T07:27:22.101 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:27:22.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/9/new.json 2024-09-16T07:27:22.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDZ3edmle3aHhAACvqAx2vb5FhP4LoamJQhMA== --osd-uuid 1219e850-bad2-4898-8efe-09fa3f2b1315 2024-09-16T07:27:22.150 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:27:22.148+0000 7f8f08640880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:27:22.154 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:27:22.152+0000 7f8f08640880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:27:22.156 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:27:22.152+0000 7f8f08640880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:27:22.157 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:27:22.156+0000 7f8f08640880 -1 bluestore(td/test-erasure-code/9/block) _read_bdev_label failed to open td/test-erasure-code/9/block: (2) No such file or directory 2024-09-16T07:27:22.157 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:27:22.156+0000 7f8f08640880 -1 bluestore(td/test-erasure-code/9/block) _read_bdev_label failed to open td/test-erasure-code/9/block: (2) No such file or directory 2024-09-16T07:27:22.157 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:27:22.156+0000 7f8f08640880 -1 bluestore(td/test-erasure-code/9/block) _read_bdev_label failed to open td/test-erasure-code/9/block: (2) No such file or directory 2024-09-16T07:27:22.327 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:27:22.324+0000 7f8f08640880 -1 bluestore(td/test-erasure-code/9) _read_fsid unparsable uuid 2024-09-16T07:27:28.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/9/keyring 2024-09-16T07:27:28.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:27:28.700 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd9 key to auth repository 2024-09-16T07:27:28.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-09-16T07:27:28.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:27:29.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-09-16T07:27:29.270 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.9 2024-09-16T07:27:29.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:27:29.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:27:29.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:27:29.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:27:29.302 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:27:29.300+0000 7fdb8a1f8880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:27:29.305 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:27:29.304+0000 7fdb8a1f8880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:27:29.307 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:27:29.304+0000 7fdb8a1f8880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:27:29.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-09-16T07:27:29.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:27:29.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-09-16T07:27:29.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:27:29.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:27:29.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:29.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:27:29.742 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:27:29.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:29.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-16T07:27:29.880 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:27:29.876+0000 7fdb8a1f8880 -1 Falling back to public interface 2024-09-16T07:27:30.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:31.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:31.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:31.232 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:27:31.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:27:31.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:31.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-16T07:27:31.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:32.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:32.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:32.650 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:27:32.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:27:32.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:32.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-16T07:27:32.705 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:27:32.700+0000 7fdb8a1f8880 -1 osd.9 0 log_to_monitors true 2024-09-16T07:27:33.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:34.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:34.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:34.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:27:34.159 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:27:34.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:34.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-16T07:27:34.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:35.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:35.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:35.630 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:27:35.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:27:35.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:35.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-16T07:27:36.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:37.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:37.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:37.060 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:27:37.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:27:37.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:37.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-16T07:27:37.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:38.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:38.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:38.612 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:27:38.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:27:38.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:38.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-16T07:27:39.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:40.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:40.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:40.052 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:27:40.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:27:40.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:40.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-16T07:27:40.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:41.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:41.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:41.591 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:27:41.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:27:41.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:41.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-16T07:27:42.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:43.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:43.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:43.030 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:27:43.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:27:43.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:43.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-16T07:27:43.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:44.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:44.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:44.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:27:44.604 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:27:44.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:44.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-16T07:27:45.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:46.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:46.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:46.281 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:27:46.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:27:46.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:46.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-16T07:27:46.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:47.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:47.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:47.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:27:47.934 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:27:47.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:47.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-16T07:27:48.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:49.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:49.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:49.699 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:27:49.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:27:49.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:49.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-16T07:27:50.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:51.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:51.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:51.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:27:51.682 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:27:51.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:51.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-16T07:27:52.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:53.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:53.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:53.387 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:27:53.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:27:53.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:53.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-16T07:27:54.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:55.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:55.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:55.514 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:27:55.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:27:55.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:55.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-16T07:27:56.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:27:57.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:27:57.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:27:57.071 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:27:57.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:27:57.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:27:57.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-16T07:27:57.564 INFO:tasks.workunit.client.0.smithi162.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/1407999799,v1:127.0.0.1:6875/1407999799] [v2:127.0.0.1:6876/1407999799,v1:127.0.0.1:6877/1407999799] exists,up 1219e850-bad2-4898-8efe-09fa3f2b1315 2024-09-16T07:27:57.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:27:57.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:27:57.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:27:57.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-16T07:27:57.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 10 2024-09-16T07:27:57.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-16T07:27:57.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:27:57.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=10 2024-09-16T07:27:57.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:27:57.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/10 2024-09-16T07:27:57.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-16T07:27:57.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:27:57.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:27:57.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:27:57.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/10' 2024-09-16T07:27:57.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/10/journal' 2024-09-16T07:27:57.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:27:57.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:27:57.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-16T07:27:57.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:27:57.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:27:57.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:27:57.568 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:27:57.568 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:27:57.568 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:27:57.569 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:27:57.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:27:57.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:27:57.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:27:57.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:27:57.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-16T07:27:57.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-16T07:27:57.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:27:57.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:27:57.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:27:57.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:27:57.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:27:57.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:27:57.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/10 2024-09-16T07:27:57.571 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:27:57.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6dfed359-0270-4e6d-9ee0-e34a075aeb36 2024-09-16T07:27:57.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd10 6dfed359-0270-4e6d-9ee0-e34a075aeb36' 2024-09-16T07:27:57.572 INFO:tasks.workunit.client.0.smithi162.stdout:add osd10 6dfed359-0270-4e6d-9ee0-e34a075aeb36 2024-09-16T07:27:57.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:27:57.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD93edmNEjSIxAAxBcd4NgLo9fbS6dLJnxl1Q== 2024-09-16T07:27:57.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD93edmNEjSIxAAxBcd4NgLo9fbS6dLJnxl1Q=="}' 2024-09-16T07:27:57.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6dfed359-0270-4e6d-9ee0-e34a075aeb36 -i td/test-erasure-code/10/new.json 2024-09-16T07:27:58.371 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:27:58.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/10/new.json 2024-09-16T07:27:58.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 10 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD93edmNEjSIxAAxBcd4NgLo9fbS6dLJnxl1Q== --osd-uuid 6dfed359-0270-4e6d-9ee0-e34a075aeb36 2024-09-16T07:27:58.430 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:27:58.427+0000 7f7ce6854880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:27:58.443 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:27:58.439+0000 7f7ce6854880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:27:58.445 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:27:58.443+0000 7f7ce6854880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:27:58.446 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:27:58.443+0000 7f7ce6854880 -1 bluestore(td/test-erasure-code/10/block) _read_bdev_label failed to open td/test-erasure-code/10/block: (2) No such file or directory 2024-09-16T07:27:58.446 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:27:58.443+0000 7f7ce6854880 -1 bluestore(td/test-erasure-code/10/block) _read_bdev_label failed to open td/test-erasure-code/10/block: (2) No such file or directory 2024-09-16T07:27:58.446 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:27:58.443+0000 7f7ce6854880 -1 bluestore(td/test-erasure-code/10/block) _read_bdev_label failed to open td/test-erasure-code/10/block: (2) No such file or directory 2024-09-16T07:27:58.560 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:27:58.555+0000 7f7ce6854880 -1 bluestore(td/test-erasure-code/10) _read_fsid unparsable uuid 2024-09-16T07:28:05.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/10/keyring 2024-09-16T07:28:05.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:28:05.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd10 key to auth repository 2024-09-16T07:28:05.196 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd10 key to auth repository 2024-09-16T07:28:05.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/10/keyring auth add osd.10 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:28:05.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.10 2024-09-16T07:28:05.776 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.10 2024-09-16T07:28:05.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 10 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:28:05.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:28:05.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:28:05.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:28:05.810 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:28:05.807+0000 7f019c445880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:28:05.819 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:28:05.815+0000 7f019c445880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:28:05.820 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:28:05.815+0000 7f019c445880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:28:06.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 10 2024-09-16T07:28:06.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:28:06.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=10 2024-09-16T07:28:06.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:28:06.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:28:06.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:28:06.267 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:28:06.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:28:06.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:28:06.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-16T07:28:06.391 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:28:06.387+0000 7f019c445880 -1 Falling back to public interface 2024-09-16T07:28:06.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:28:07.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:28:07.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:28:07.770 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:28:07.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:28:07.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:28:07.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-16T07:28:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:28:08.693 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:28:08.691+0000 7f019c445880 -1 osd.10 0 log_to_monitors true 2024-09-16T07:28:09.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:28:09.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:28:09.202 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:28:09.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:28:09.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:28:09.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-16T07:28:09.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:28:10.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:28:10.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:28:10.720 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:28:10.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:28:10.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:28:10.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-16T07:28:11.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:28:12.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:28:12.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:28:12.198 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:28:12.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:28:12.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:28:12.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-16T07:28:12.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:28:13.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:28:13.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:28:13.629 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:28:13.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:28:13.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:28:13.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-16T07:28:14.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:28:15.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:28:15.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:28:15.069 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:28:15.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:28:15.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:28:15.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-16T07:28:15.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:28:16.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:28:16.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:28:16.574 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:28:16.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:28:16.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:28:16.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-16T07:28:17.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:28:18.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:28:18.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:28:18.027 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:28:18.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:28:18.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:28:18.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-16T07:28:18.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:28:19.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:28:19.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:28:19.476 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:28:19.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:28:19.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:28:19.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-16T07:28:19.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:28:20.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:28:20.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:28:20.978 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:28:20.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:28:20.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:28:20.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-16T07:28:21.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:28:22.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:28:22.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:28:22.427 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:28:22.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:28:22.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:28:22.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-16T07:28:23.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:28:24.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:28:24.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:28:24.345 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:28:24.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:28:24.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:28:24.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-16T07:28:25.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:28:26.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:28:26.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:28:26.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:28:26.183 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:28:26.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:28:26.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-16T07:28:27.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:28:28.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:28:28.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:28:28.073 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:28:28.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:28:28.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:28:28.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-16T07:28:28.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:28:29.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:28:29.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:28:29.958 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:28:29.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:28:29.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-16T07:28:29.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:28:30.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:28:31.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:28:31.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:28:31.748 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:28:31.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:28:31.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:28:31.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-16T07:28:32.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:28:33.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:28:33.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:28:33.541 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:28:33.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:28:33.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:28:33.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-16T07:28:34.217 INFO:tasks.workunit.client.0.smithi162.stdout:osd.10 up in weight 1 up_from 55 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6882/1219683675,v1:127.0.0.1:6883/1219683675] [v2:127.0.0.1:6884/1219683675,v1:127.0.0.1:6885/1219683675] exists,up 6dfed359-0270-4e6d-9ee0-e34a075aeb36 2024-09-16T07:28:34.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:28:34.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:28:34.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:28:34.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:39: run: create_rbd_pool 2024-09-16T07:28:34.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T07:28:34.648 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' does not exist 2024-09-16T07:28:34.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T07:28:34.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T07:28:35.342 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-16T07:28:35.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T07:28:36.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T07:28:39.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:40: run: wait_for_clean 2024-09-16T07:28:39.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T07:28:39.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T07:28:39.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T07:28:39.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T07:28:39.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T07:28:39.265 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T07:28:39.266 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T07:28:39.266 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T07:28:39.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T07:28:39.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T07:28:39.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T07:28:39.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T07:28:39.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T07:28:39.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T07:28:39.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T07:28:39.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T07:28:39.893 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:28:39.893 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:28:39.893 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:28:39.893 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:28:39.893 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:28:39.893 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-16T07:28:39.893 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-16T07:28:39.894 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-09-16T07:28:39.894 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-16T07:28:39.894 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-16T07:28:39.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T07:28:39.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:28:39.894 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T07:28:40.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836564 2024-09-16T07:28:40.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836564 2024-09-16T07:28:40.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836564' 2024-09-16T07:28:40.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:28:40.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T07:28:40.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673038 2024-09-16T07:28:40.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673038 2024-09-16T07:28:40.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836564 1-42949673038' 2024-09-16T07:28:40.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:28:40.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T07:28:40.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509513 2024-09-16T07:28:40.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509513 2024-09-16T07:28:40.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509513' 2024-09-16T07:28:40.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:28:40.434 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T07:28:40.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345986 2024-09-16T07:28:40.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345986 2024-09-16T07:28:40.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509513 3-85899345986' 2024-09-16T07:28:40.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:28:40.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T07:28:40.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182463 2024-09-16T07:28:40.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182463 2024-09-16T07:28:40.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509513 3-85899345986 4-107374182463' 2024-09-16T07:28:40.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:28:40.743 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T07:28:40.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=128849018930 2024-09-16T07:28:40.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 128849018930 2024-09-16T07:28:40.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509513 3-85899345986 4-107374182463 5-128849018930' 2024-09-16T07:28:40.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:28:40.897 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-16T07:28:41.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=150323855410 2024-09-16T07:28:41.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 150323855410 2024-09-16T07:28:41.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509513 3-85899345986 4-107374182463 5-128849018930 6-150323855410' 2024-09-16T07:28:41.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:28:41.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-16T07:28:41.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=171798691880 2024-09-16T07:28:41.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 171798691880 2024-09-16T07:28:41.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509513 3-85899345986 4-107374182463 5-128849018930 6-150323855410 7-171798691880' 2024-09-16T07:28:41.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:28:41.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-16T07:28:41.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=193273528351 2024-09-16T07:28:41.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 193273528351 2024-09-16T07:28:41.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509513 3-85899345986 4-107374182463 5-128849018930 6-150323855410 7-171798691880 8-193273528351' 2024-09-16T07:28:41.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:28:41.352 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-16T07:28:41.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=214748364826 2024-09-16T07:28:41.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 214748364826 2024-09-16T07:28:41.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509513 3-85899345986 4-107374182463 5-128849018930 6-150323855410 7-171798691880 8-193273528351 9-214748364826' 2024-09-16T07:28:41.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:28:41.503 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-16T07:28:41.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=236223201299 2024-09-16T07:28:41.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 236223201299 2024-09-16T07:28:41.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509513 3-85899345986 4-107374182463 5-128849018930 6-150323855410 7-171798691880 8-193273528351 9-214748364826 10-236223201299' 2024-09-16T07:28:41.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:28:41.661 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836564 2024-09-16T07:28:41.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:28:41.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T07:28:41.664 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836564 2024-09-16T07:28:41.664 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:28:41.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836564 2024-09-16T07:28:41.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836564' 2024-09-16T07:28:41.666 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836564 2024-09-16T07:28:41.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:28:42.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836565 -lt 21474836564 2024-09-16T07:28:42.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:28:42.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:28:42.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673038 2024-09-16T07:28:42.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T07:28:42.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673038 2024-09-16T07:28:42.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:28:42.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673038 2024-09-16T07:28:42.103 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949673038 2024-09-16T07:28:42.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673038' 2024-09-16T07:28:42.104 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T07:28:42.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673039 -lt 42949673038 2024-09-16T07:28:42.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:28:42.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509513 2024-09-16T07:28:42.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:28:42.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T07:28:42.543 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509513 2024-09-16T07:28:42.543 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:28:42.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509513 2024-09-16T07:28:42.546 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509513 2024-09-16T07:28:42.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509513' 2024-09-16T07:28:42.546 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T07:28:43.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509514 -lt 64424509513 2024-09-16T07:28:43.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:28:43.046 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345986 2024-09-16T07:28:43.046 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:28:43.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T07:28:43.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345986 2024-09-16T07:28:43.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:28:43.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345986 2024-09-16T07:28:43.051 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 85899345986 2024-09-16T07:28:43.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345986' 2024-09-16T07:28:43.051 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T07:28:43.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345986 -lt 85899345986 2024-09-16T07:28:43.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:28:43.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182463 2024-09-16T07:28:43.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:28:43.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T07:28:43.555 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:28:43.555 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182463 2024-09-16T07:28:43.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182463 2024-09-16T07:28:43.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182463' 2024-09-16T07:28:43.556 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 107374182463 2024-09-16T07:28:43.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T07:28:44.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182464 -lt 107374182463 2024-09-16T07:28:44.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:28:44.044 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-128849018930 2024-09-16T07:28:44.044 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:28:44.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T07:28:44.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-128849018930 2024-09-16T07:28:44.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:28:44.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=128849018930 2024-09-16T07:28:44.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 128849018930' 2024-09-16T07:28:44.049 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 128849018930 2024-09-16T07:28:44.050 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T07:28:44.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 128849018931 -lt 128849018930 2024-09-16T07:28:44.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:28:44.813 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 6-150323855410 2024-09-16T07:28:44.813 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:28:44.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=6 2024-09-16T07:28:44.815 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 6-150323855410 2024-09-16T07:28:44.816 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:28:44.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=150323855410 2024-09-16T07:28:44.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.6 seq 150323855410' 2024-09-16T07:28:44.817 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 150323855410 2024-09-16T07:28:44.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-16T07:28:45.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 150323855410 -lt 150323855410 2024-09-16T07:28:45.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:28:45.228 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 7-171798691880 2024-09-16T07:28:45.228 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:28:45.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=7 2024-09-16T07:28:45.231 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 7-171798691880 2024-09-16T07:28:45.232 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:28:45.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=171798691880 2024-09-16T07:28:45.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.7 seq 171798691880' 2024-09-16T07:28:45.233 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.7 seq 171798691880 2024-09-16T07:28:45.234 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-16T07:28:45.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 171798691880 -lt 171798691880 2024-09-16T07:28:45.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:28:45.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 8-193273528351 2024-09-16T07:28:45.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:28:45.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=8 2024-09-16T07:28:45.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 8-193273528351 2024-09-16T07:28:45.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:28:45.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=193273528351 2024-09-16T07:28:45.657 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.8 seq 193273528351 2024-09-16T07:28:45.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.8 seq 193273528351' 2024-09-16T07:28:45.658 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-16T07:28:46.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 193273528352 -lt 193273528351 2024-09-16T07:28:46.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:28:46.062 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 9-214748364826 2024-09-16T07:28:46.062 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:28:46.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=9 2024-09-16T07:28:46.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 9-214748364826 2024-09-16T07:28:46.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:28:46.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=214748364826 2024-09-16T07:28:46.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.9 seq 214748364826' 2024-09-16T07:28:46.067 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.9 seq 214748364826 2024-09-16T07:28:46.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-16T07:28:46.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 214748364827 -lt 214748364826 2024-09-16T07:28:46.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:28:46.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 10-236223201299 2024-09-16T07:28:46.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:28:46.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=10 2024-09-16T07:28:46.619 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 10-236223201299 2024-09-16T07:28:46.619 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:28:46.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=236223201299 2024-09-16T07:28:46.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.10 seq 236223201299' 2024-09-16T07:28:46.621 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.10 seq 236223201299 2024-09-16T07:28:46.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-16T07:28:47.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 236223201300 -lt 236223201299 2024-09-16T07:28:47.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T07:28:47.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:28:47.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:28:47.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T07:28:47.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T07:28:47.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T07:28:47.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T07:28:47.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T07:28:47.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T07:28:47.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T07:28:47.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T07:28:48.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T07:28:48.161 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T07:28:48.161 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:28:48.162 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:28:48.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T07:28:48.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T07:28:48.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T07:28:48.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: get_asok_path osd.0 2024-09-16T07:28:48.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T07:28:48.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T07:28:48.696 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:28:48.696 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:28:48.697 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:28:48.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46203/ceph-osd.0.asok 2024-09-16T07:28:48.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: CEPH_ARGS= 2024-09-16T07:28:48.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: ceph --admin-daemon /tmp/ceph-asok.46203/ceph-osd.0.asok log flush 2024-09-16T07:28:48.780 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:28:48.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:43: run: grep 'load: jerasure.*lrc' td/test-erasure-code/osd.0.log 2024-09-16T07:28:48.794 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:22:10.004+0000 7f9417297880 0 load: jerasure load: lrc 2024-09-16T07:28:48.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:44: run: create_erasure_coded_pool ecpool 2024-09-16T07:28:48.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:56: create_erasure_coded_pool: local poolname=ecpool 2024-09-16T07:28:48.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:58: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd 2024-09-16T07:28:49.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:60: create_erasure_coded_pool: create_pool ecpool 12 12 erasure myprofile 2024-09-16T07:28:49.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 12 12 erasure myprofile 2024-09-16T07:28:50.305 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'ecpool' already exists 2024-09-16T07:28:50.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T07:28:51.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:62: create_erasure_coded_pool: wait_for_clean 2024-09-16T07:28:51.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T07:28:51.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T07:28:51.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T07:28:51.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T07:28:51.317 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T07:28:51.318 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T07:28:51.318 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T07:28:51.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T07:28:51.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T07:28:51.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T07:28:51.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T07:28:51.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T07:28:51.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T07:28:51.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T07:28:51.523 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T07:28:52.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T07:28:52.482 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:28:52.482 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:28:52.482 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:28:52.483 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:28:52.483 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:28:52.483 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-16T07:28:52.483 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-16T07:28:52.483 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-09-16T07:28:52.483 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-16T07:28:52.483 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-16T07:28:52.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T07:28:52.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:28:52.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T07:28:52.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836569 2024-09-16T07:28:52.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836569 2024-09-16T07:28:52.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836569' 2024-09-16T07:28:52.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:28:52.891 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T07:28:53.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673043 2024-09-16T07:28:53.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673043 2024-09-16T07:28:53.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836569 1-42949673043' 2024-09-16T07:28:53.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:28:53.308 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T07:28:53.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509518 2024-09-16T07:28:53.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509518 2024-09-16T07:28:53.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509518' 2024-09-16T07:28:53.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:28:53.781 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T07:28:54.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345991 2024-09-16T07:28:54.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345991 2024-09-16T07:28:54.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509518 3-85899345991' 2024-09-16T07:28:54.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:28:54.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T07:28:54.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182468 2024-09-16T07:28:54.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182468 2024-09-16T07:28:54.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509518 3-85899345991 4-107374182468' 2024-09-16T07:28:54.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:28:54.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T07:28:54.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=128849018935 2024-09-16T07:28:54.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 128849018935 2024-09-16T07:28:54.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509518 3-85899345991 4-107374182468 5-128849018935' 2024-09-16T07:28:54.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:28:54.424 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-16T07:28:54.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=150323855415 2024-09-16T07:28:54.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 150323855415 2024-09-16T07:28:54.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509518 3-85899345991 4-107374182468 5-128849018935 6-150323855415' 2024-09-16T07:28:54.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:28:54.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-16T07:28:54.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=171798691884 2024-09-16T07:28:54.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 171798691884 2024-09-16T07:28:54.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509518 3-85899345991 4-107374182468 5-128849018935 6-150323855415 7-171798691884' 2024-09-16T07:28:54.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:28:54.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-16T07:28:54.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=193273528355 2024-09-16T07:28:54.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 193273528355 2024-09-16T07:28:54.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509518 3-85899345991 4-107374182468 5-128849018935 6-150323855415 7-171798691884 8-193273528355' 2024-09-16T07:28:54.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:28:54.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-16T07:28:55.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=214748364831 2024-09-16T07:28:55.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 214748364831 2024-09-16T07:28:55.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509518 3-85899345991 4-107374182468 5-128849018935 6-150323855415 7-171798691884 8-193273528355 9-214748364831' 2024-09-16T07:28:55.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:28:55.107 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-16T07:28:55.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=236223201304 2024-09-16T07:28:55.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 236223201304 2024-09-16T07:28:55.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509518 3-85899345991 4-107374182468 5-128849018935 6-150323855415 7-171798691884 8-193273528355 9-214748364831 10-236223201304' 2024-09-16T07:28:55.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:28:55.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836569 2024-09-16T07:28:55.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:28:55.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T07:28:55.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:28:55.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836569 2024-09-16T07:28:55.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836569 2024-09-16T07:28:55.264 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836569 2024-09-16T07:28:55.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836569' 2024-09-16T07:28:55.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:28:55.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836569 -lt 21474836569 2024-09-16T07:28:55.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:28:55.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673043 2024-09-16T07:28:55.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:28:55.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T07:28:55.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673043 2024-09-16T07:28:55.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:28:55.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673043 2024-09-16T07:28:55.755 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949673043 2024-09-16T07:28:55.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673043' 2024-09-16T07:28:55.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T07:28:56.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673043 -lt 42949673043 2024-09-16T07:28:56.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:28:56.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:28:56.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509518 2024-09-16T07:28:56.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T07:28:56.255 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509518 2024-09-16T07:28:56.256 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:28:56.257 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509518 2024-09-16T07:28:56.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509518 2024-09-16T07:28:56.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509518' 2024-09-16T07:28:56.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T07:28:56.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509518 -lt 64424509518 2024-09-16T07:28:56.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:28:56.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345991 2024-09-16T07:28:56.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:28:56.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T07:28:56.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345991 2024-09-16T07:28:56.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:28:56.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345991 2024-09-16T07:28:56.694 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 85899345991 2024-09-16T07:28:56.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345991' 2024-09-16T07:28:56.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T07:28:57.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345991 -lt 85899345991 2024-09-16T07:28:57.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:28:57.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182468 2024-09-16T07:28:57.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:28:57.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T07:28:57.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182468 2024-09-16T07:28:57.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:28:57.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182468 2024-09-16T07:28:57.199 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 107374182468 2024-09-16T07:28:57.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182468' 2024-09-16T07:28:57.200 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T07:28:57.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182468 -lt 107374182468 2024-09-16T07:28:57.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:28:57.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-128849018935 2024-09-16T07:28:57.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:28:57.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T07:28:57.657 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-128849018935 2024-09-16T07:28:57.658 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:28:57.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=128849018935 2024-09-16T07:28:57.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 128849018935' 2024-09-16T07:28:57.658 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 128849018935 2024-09-16T07:28:57.658 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T07:28:58.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 128849018936 -lt 128849018935 2024-09-16T07:28:58.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:28:58.104 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 6-150323855415 2024-09-16T07:28:58.104 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:28:58.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=6 2024-09-16T07:28:58.107 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 6-150323855415 2024-09-16T07:28:58.107 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:28:58.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=150323855415 2024-09-16T07:28:58.109 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 150323855415 2024-09-16T07:28:58.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.6 seq 150323855415' 2024-09-16T07:28:58.110 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-16T07:28:58.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 150323855415 -lt 150323855415 2024-09-16T07:28:58.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:28:58.548 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 7-171798691884 2024-09-16T07:28:58.548 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:28:58.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=7 2024-09-16T07:28:58.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 7-171798691884 2024-09-16T07:28:58.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:28:58.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=171798691884 2024-09-16T07:28:58.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.7 seq 171798691884' 2024-09-16T07:28:58.553 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.7 seq 171798691884 2024-09-16T07:28:58.553 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-16T07:28:58.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 171798691885 -lt 171798691884 2024-09-16T07:28:58.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:28:58.967 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 8-193273528355 2024-09-16T07:28:58.967 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:28:58.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=8 2024-09-16T07:28:58.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 8-193273528355 2024-09-16T07:28:58.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:28:58.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=193273528355 2024-09-16T07:28:58.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.8 seq 193273528355' 2024-09-16T07:28:58.972 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.8 seq 193273528355 2024-09-16T07:28:58.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-16T07:28:59.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 193273528356 -lt 193273528355 2024-09-16T07:28:59.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:28:59.416 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 9-214748364831 2024-09-16T07:28:59.416 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:28:59.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=9 2024-09-16T07:28:59.420 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 9-214748364831 2024-09-16T07:28:59.420 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:28:59.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=214748364831 2024-09-16T07:28:59.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.9 seq 214748364831' 2024-09-16T07:28:59.422 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.9 seq 214748364831 2024-09-16T07:28:59.422 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-16T07:28:59.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 214748364832 -lt 214748364831 2024-09-16T07:28:59.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:28:59.881 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 10-236223201304 2024-09-16T07:28:59.881 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:28:59.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=10 2024-09-16T07:28:59.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 10-236223201304 2024-09-16T07:28:59.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:28:59.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=236223201304 2024-09-16T07:28:59.886 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.10 seq 236223201304 2024-09-16T07:28:59.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.10 seq 236223201304' 2024-09-16T07:28:59.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-16T07:29:00.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 236223201305 -lt 236223201304 2024-09-16T07:29:00.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T07:29:00.386 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:29:00.386 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:29:00.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-16T07:29:00.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T07:29:00.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T07:29:00.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T07:29:00.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T07:29:00.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T07:29:00.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T07:29:00.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T07:29:01.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-16T07:29:01.435 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T07:29:01.436 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:29:01.436 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:29:01.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-16T07:29:01.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T07:29:01.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T07:29:01.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: set 2024-09-16T07:29:01.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T07:29:01.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: local 'funcs=TEST_alignment_constraints 2024-09-16T07:29:01.984 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_chunk_mapping 2024-09-16T07:29:01.984 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_rados_put_get_isa 2024-09-16T07:29:01.984 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_rados_put_get_jerasure 2024-09-16T07:29:01.984 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_rados_put_get_lrc_advanced 2024-09-16T07:29:01.984 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_rados_put_get_lrc_kml 2024-09-16T07:29:01.985 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_rados_put_get_shec' 2024-09-16T07:29:01.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-16T07:29:01.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_alignment_constraints td/test-erasure-code 2024-09-16T07:29:01.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: TEST_alignment_constraints: local payload=ABC 2024-09-16T07:29:01.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:254: TEST_alignment_constraints: echo ABC 2024-09-16T07:29:01.985 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:260: TEST_alignment_constraints: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-16T07:29:02.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:260: TEST_alignment_constraints: local stripe_unit=4096 2024-09-16T07:29:02.005 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: ceph osd erasure-code-profile get myprofile 2024-09-16T07:29:02.005 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: grep k= 2024-09-16T07:29:02.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: eval local k=2 2024-09-16T07:29:02.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: local k=2 2024-09-16T07:29:02.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:262: TEST_alignment_constraints: local block_size=8191 2024-09-16T07:29:02.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:263: TEST_alignment_constraints: dd if=/dev/zero of=td/test-erasure-code/ORIGINAL bs=8191 count=2 2024-09-16T07:29:02.495 INFO:tasks.workunit.client.0.smithi162.stderr:2+0 records in 2024-09-16T07:29:02.495 INFO:tasks.workunit.client.0.smithi162.stderr:2+0 records out 2024-09-16T07:29:02.495 INFO:tasks.workunit.client.0.smithi162.stderr:16382 bytes (16 kB, 16 KiB) copied, 0.00027375 s, 59.8 MB/s 2024-09-16T07:29:02.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:264: TEST_alignment_constraints: rados --block-size=8191 --pool ecpool put UNALIGNED td/test-erasure-code/ORIGINAL 2024-09-16T07:29:02.562 INFO:tasks.workunit.client.0.smithi162.stderr:INFO: op_size has been rounded to 8192 2024-09-16T07:29:02.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: TEST_alignment_constraints: rm td/test-erasure-code/ORIGINAL 2024-09-16T07:29:02.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-16T07:29:02.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_chunk_mapping td/test-erasure-code 2024-09-16T07:29:02.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:304: TEST_chunk_mapping: local dir=td/test-erasure-code 2024-09-16T07:29:02.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:311: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code ecpool 0 1 2024-09-16T07:29:02.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:281: verify_chunk_mapping: local dir=td/test-erasure-code 2024-09-16T07:29:02.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: local poolname=ecpool 2024-09-16T07:29:02.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:283: verify_chunk_mapping: local first=0 2024-09-16T07:29:02.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:284: verify_chunk_mapping: local second=1 2024-09-16T07:29:02.976 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-09-16T07:29:02.977 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-16T07:29:03.000 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-09-16T07:29:03.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 FIRSTecpool 2024-09-16T07:29:03.001 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-09-16T07:29:03.002 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-16T07:29:03.023 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-09-16T07:29:03.024 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 SECONDecpool 2024-09-16T07:29:03.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: local 'payload= FIRSTecpool SECONDecpool' 2024-09-16T07:29:03.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: echo -n ' FIRSTecpool SECONDecpool' 2024-09-16T07:29:03.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: rados --pool ecpool put SOMETHINGecpool td/test-erasure-code/ORIGINAL 2024-09-16T07:29:03.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:290: verify_chunk_mapping: rados --pool ecpool get SOMETHINGecpool td/test-erasure-code/COPY 2024-09-16T07:29:03.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: osds=($(get_osds $poolname SOMETHING$poolname)) 2024-09-16T07:29:03.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2024-09-16T07:29:03.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-09-16T07:29:03.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGecpool 2024-09-16T07:29:03.230 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2024-09-16T07:29:03.230 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:29:03.693 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T07:29:03.694 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:29:03.694 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-16T07:29:03.694 INFO:tasks.workunit.client.0.smithi162.stderr:0' 2024-09-16T07:29:03.694 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 2024-09-16T07:29:03.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: local -a osds 2024-09-16T07:29:03.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i = 0 )) 2024-09-16T07:29:03.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-09-16T07:29:03.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.3 flush_journal 2024-09-16T07:29:03.811 INFO:tasks.workunit.client.0.smithi162.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-16T07:29:03.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-16T07:29:03.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-09-16T07:29:03.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.5 flush_journal 2024-09-16T07:29:03.931 INFO:tasks.workunit.client.0.smithi162.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-16T07:29:03.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-16T07:29:03.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-09-16T07:29:03.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2024-09-16T07:29:04.057 INFO:tasks.workunit.client.0.smithi162.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-16T07:29:04.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-16T07:29:04.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-09-16T07:29:04.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.0 flush_journal 2024-09-16T07:29:04.183 INFO:tasks.workunit.client.0.smithi162.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-16T07:29:04.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-16T07:29:04.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-09-16T07:29:04.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-16T07:29:04.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: rm td/test-erasure-code/COPY 2024-09-16T07:29:04.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: osds=($(get_osds $poolname SOMETHING$poolname)) 2024-09-16T07:29:04.204 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2024-09-16T07:29:04.204 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-09-16T07:29:04.204 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGecpool 2024-09-16T07:29:04.205 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2024-09-16T07:29:04.205 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:29:04.644 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T07:29:04.644 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:29:04.644 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-16T07:29:04.644 INFO:tasks.workunit.client.0.smithi162.stderr:0' 2024-09-16T07:29:04.644 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 2024-09-16T07:29:04.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: local -a osds 2024-09-16T07:29:04.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: grep --quiet FIRSTecpool 2024-09-16T07:29:04.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: objectstore_tool td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-09-16T07:29:04.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-09-16T07:29:04.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T07:29:04.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-16T07:29:04.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T07:29:04.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-09-16T07:29:04.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-09-16T07:29:04.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T07:29:04.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-16T07:29:04.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T07:29:04.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.3 2024-09-16T07:29:04.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:29:04.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:29:04.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:29:04.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:29:04.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:29:05.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:29:05.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-09-16T07:29:05.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-09-16T07:29:05.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T07:29:05.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-16T07:29:05.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T07:29:05.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/3 2024-09-16T07:29:05.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/3 SOMETHINGecpool get-bytes 2024-09-16T07:29:07.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 3 2024-09-16T07:29:07.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-09-16T07:29:07.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T07:29:07.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-16T07:29:07.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T07:29:07.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/3 2024-09-16T07:29:07.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-16T07:29:07.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:29:07.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:29:07.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:29:07.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2024-09-16T07:29:07.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2024-09-16T07:29:07.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T07:29:07.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T07:29:07.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-16T07:29:07.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T07:29:07.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:29:07.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:29:07.754 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:29:07.754 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:29:07.754 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:29:07.754 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:29:07.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:29:07.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:29:07.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-16T07:29:07.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-16T07:29:07.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:29:07.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:29:07.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:29:07.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:29:07.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T07:29:07.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T07:29:07.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/3 2024-09-16T07:29:07.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-16T07:29:07.755 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.3 2024-09-16T07:29:07.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:29:07.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/3/whoami 2024-09-16T07:29:07.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-16T07:29:07.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T07:29:07.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T07:29:07.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T07:29:07.793 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:29:07.794+0000 7f45abab2880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:29:07.831 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:29:07.830+0000 7f45abab2880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:29:07.834 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:29:07.834+0000 7f45abab2880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:29:08.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-16T07:29:08.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:29:08.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T07:29:08.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:29:08.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:29:08.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:29:08.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:29:08.289 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T07:29:08.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:29:08.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:29:08.403 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:29:08.402+0000 7f45abab2880 -1 Falling back to public interface 2024-09-16T07:29:08.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:29:09.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:29:09.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:29:09.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:29:09.718 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:29:09.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:29:09.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:29:10.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:29:10.828 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:29:10.826+0000 7f45abab2880 -1 osd.3 74 log_to_monitors true 2024-09-16T07:29:11.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:29:11.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:29:11.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:29:11.209 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:29:11.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:29:11.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:29:12.030 INFO:tasks.workunit.client.0.smithi162.stderr:osd.3 up in weight 1 up_from 78 up_thru 78 down_at 75 last_clean_interval [20,74) [v2:127.0.0.1:6826/3280939065,v1:127.0.0.1:6827/3280939065] [v2:127.0.0.1:6828/3280939065,v1:127.0.0.1:6829/3280939065] exists,up 2126ebea-dac7-4748-8e6b-84428d3991be 2024-09-16T07:29:12.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:29:12.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:29:12.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:29:12.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-16T07:29:12.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T07:29:12.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T07:29:12.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T07:29:12.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T07:29:12.031 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T07:29:12.032 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T07:29:12.032 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T07:29:12.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T07:29:12.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T07:29:12.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T07:29:12.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T07:29:12.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T07:29:12.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T07:29:12.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T07:29:12.232 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T07:29:12.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T07:29:12.860 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:29:12.860 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:29:12.860 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:29:12.860 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:29:12.860 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:29:12.860 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-16T07:29:12.860 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-16T07:29:12.860 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-09-16T07:29:12.860 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-16T07:29:12.860 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-16T07:29:12.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T07:29:12.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:12.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T07:29:13.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836575 2024-09-16T07:29:13.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836575 2024-09-16T07:29:13.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836575' 2024-09-16T07:29:13.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:13.023 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T07:29:13.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673049 2024-09-16T07:29:13.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673049 2024-09-16T07:29:13.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836575 1-42949673049' 2024-09-16T07:29:13.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:13.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T07:29:13.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509524 2024-09-16T07:29:13.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509524 2024-09-16T07:29:13.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836575 1-42949673049 2-64424509524' 2024-09-16T07:29:13.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:13.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T07:29:13.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=335007449091 2024-09-16T07:29:13.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 335007449091 2024-09-16T07:29:13.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836575 1-42949673049 2-64424509524 3-335007449091' 2024-09-16T07:29:13.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:13.523 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T07:29:13.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182474 2024-09-16T07:29:13.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182474 2024-09-16T07:29:13.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836575 1-42949673049 2-64424509524 3-335007449091 4-107374182474' 2024-09-16T07:29:13.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:13.677 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T07:29:13.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=128849018941 2024-09-16T07:29:13.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 128849018941 2024-09-16T07:29:13.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836575 1-42949673049 2-64424509524 3-335007449091 4-107374182474 5-128849018941' 2024-09-16T07:29:13.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:13.843 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-16T07:29:13.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=150323855421 2024-09-16T07:29:13.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 150323855421 2024-09-16T07:29:13.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836575 1-42949673049 2-64424509524 3-335007449091 4-107374182474 5-128849018941 6-150323855421' 2024-09-16T07:29:13.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:13.989 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-16T07:29:14.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=171798691890 2024-09-16T07:29:14.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 171798691890 2024-09-16T07:29:14.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836575 1-42949673049 2-64424509524 3-335007449091 4-107374182474 5-128849018941 6-150323855421 7-171798691890' 2024-09-16T07:29:14.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:14.153 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-16T07:29:14.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=193273528361 2024-09-16T07:29:14.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 193273528361 2024-09-16T07:29:14.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836575 1-42949673049 2-64424509524 3-335007449091 4-107374182474 5-128849018941 6-150323855421 7-171798691890 8-193273528361' 2024-09-16T07:29:14.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:14.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-16T07:29:14.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=214748364837 2024-09-16T07:29:14.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 214748364837 2024-09-16T07:29:14.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836575 1-42949673049 2-64424509524 3-335007449091 4-107374182474 5-128849018941 6-150323855421 7-171798691890 8-193273528361 9-214748364837' 2024-09-16T07:29:14.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:14.492 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-16T07:29:14.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=236223201310 2024-09-16T07:29:14.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 236223201310 2024-09-16T07:29:14.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836575 1-42949673049 2-64424509524 3-335007449091 4-107374182474 5-128849018941 6-150323855421 7-171798691890 8-193273528361 9-214748364837 10-236223201310' 2024-09-16T07:29:14.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:14.645 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836575 2024-09-16T07:29:14.645 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:14.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T07:29:14.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836575 2024-09-16T07:29:14.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:14.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836575 2024-09-16T07:29:14.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836575' 2024-09-16T07:29:14.651 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836575 2024-09-16T07:29:14.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:29:15.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836575 -lt 21474836575 2024-09-16T07:29:15.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:15.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673049 2024-09-16T07:29:15.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:15.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T07:29:15.084 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673049 2024-09-16T07:29:15.085 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:15.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673049 2024-09-16T07:29:15.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673049' 2024-09-16T07:29:15.087 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949673049 2024-09-16T07:29:15.087 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T07:29:15.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673049 -lt 42949673049 2024-09-16T07:29:15.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:15.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509524 2024-09-16T07:29:15.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:15.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T07:29:15.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509524 2024-09-16T07:29:15.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:15.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509524 2024-09-16T07:29:15.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509524' 2024-09-16T07:29:15.604 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509524 2024-09-16T07:29:15.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T07:29:16.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509524 -lt 64424509524 2024-09-16T07:29:16.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:16.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-335007449091 2024-09-16T07:29:16.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:16.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T07:29:16.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-335007449091 2024-09-16T07:29:16.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:16.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=335007449091 2024-09-16T07:29:16.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 335007449091' 2024-09-16T07:29:16.044 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 335007449091 2024-09-16T07:29:16.045 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T07:29:16.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 335007449091 -lt 335007449091 2024-09-16T07:29:16.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:16.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182474 2024-09-16T07:29:16.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:16.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T07:29:16.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182474 2024-09-16T07:29:16.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:16.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182474 2024-09-16T07:29:16.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182474' 2024-09-16T07:29:16.472 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.4 seq 107374182474 2024-09-16T07:29:16.472 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T07:29:16.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182474 -lt 107374182474 2024-09-16T07:29:16.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:16.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-128849018941 2024-09-16T07:29:16.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:16.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T07:29:16.944 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-128849018941 2024-09-16T07:29:16.944 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:16.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=128849018941 2024-09-16T07:29:16.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 128849018941' 2024-09-16T07:29:16.946 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.5 seq 128849018941 2024-09-16T07:29:16.946 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T07:29:17.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 128849018941 -lt 128849018941 2024-09-16T07:29:17.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:17.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 6-150323855421 2024-09-16T07:29:17.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:17.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=6 2024-09-16T07:29:17.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 6-150323855421 2024-09-16T07:29:17.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:17.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=150323855421 2024-09-16T07:29:17.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.6 seq 150323855421' 2024-09-16T07:29:17.407 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.6 seq 150323855421 2024-09-16T07:29:17.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-16T07:29:17.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 150323855421 -lt 150323855421 2024-09-16T07:29:17.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:17.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 7-171798691890 2024-09-16T07:29:17.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:17.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=7 2024-09-16T07:29:17.857 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 7-171798691890 2024-09-16T07:29:17.857 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:17.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=171798691890 2024-09-16T07:29:17.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.7 seq 171798691890' 2024-09-16T07:29:17.859 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.7 seq 171798691890 2024-09-16T07:29:17.859 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-16T07:29:18.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 171798691891 -lt 171798691890 2024-09-16T07:29:18.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:18.283 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 8-193273528361 2024-09-16T07:29:18.283 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:18.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=8 2024-09-16T07:29:18.286 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 8-193273528361 2024-09-16T07:29:18.286 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:18.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=193273528361 2024-09-16T07:29:18.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.8 seq 193273528361' 2024-09-16T07:29:18.288 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.8 seq 193273528361 2024-09-16T07:29:18.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-16T07:29:18.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 193273528362 -lt 193273528361 2024-09-16T07:29:18.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:18.726 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 9-214748364837 2024-09-16T07:29:18.726 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:18.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=9 2024-09-16T07:29:18.729 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 9-214748364837 2024-09-16T07:29:18.729 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:18.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=214748364837 2024-09-16T07:29:18.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.9 seq 214748364837' 2024-09-16T07:29:18.732 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.9 seq 214748364837 2024-09-16T07:29:18.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-16T07:29:19.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 214748364838 -lt 214748364837 2024-09-16T07:29:19.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:19.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 10-236223201310 2024-09-16T07:29:19.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:19.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=10 2024-09-16T07:29:19.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 10-236223201310 2024-09-16T07:29:19.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:19.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=236223201310 2024-09-16T07:29:19.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.10 seq 236223201310' 2024-09-16T07:29:19.245 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.10 seq 236223201310 2024-09-16T07:29:19.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-16T07:29:19.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 236223201310 -lt 236223201310 2024-09-16T07:29:19.689 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T07:29:19.689 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:29:19.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:29:20.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-16T07:29:20.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T07:29:20.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T07:29:20.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T07:29:20.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T07:29:20.281 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T07:29:20.281 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T07:29:20.281 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T07:29:20.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-16T07:29:20.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T07:29:20.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:29:20.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:29:21.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-16T07:29:21.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T07:29:21.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T07:29:21.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: objectstore_tool td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-09-16T07:29:21.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-09-16T07:29:21.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T07:29:21.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: grep --quiet SECONDecpool 2024-09-16T07:29:21.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-09-16T07:29:21.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T07:29:21.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-09-16T07:29:21.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-09-16T07:29:21.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T07:29:21.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-09-16T07:29:21.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T07:29:21.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.5 2024-09-16T07:29:21.309 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:29:21.309 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:29:21.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:29:21.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:29:21.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:29:21.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:29:21.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-09-16T07:29:21.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-09-16T07:29:21.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T07:29:21.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-09-16T07:29:21.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T07:29:21.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/5 2024-09-16T07:29:21.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/5 SOMETHINGecpool get-bytes 2024-09-16T07:29:24.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 5 2024-09-16T07:29:24.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-09-16T07:29:24.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T07:29:24.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-09-16T07:29:24.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T07:29:24.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/5 2024-09-16T07:29:24.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-16T07:29:24.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:29:24.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:29:24.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:29:24.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2024-09-16T07:29:24.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2024-09-16T07:29:24.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T07:29:24.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T07:29:24.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-16T07:29:24.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T07:29:24.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:29:24.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:29:24.725 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:29:24.725 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:29:24.725 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:29:24.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:29:24.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:29:24.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:29:24.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-16T07:29:24.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-16T07:29:24.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:29:24.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:29:24.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:29:24.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:29:24.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T07:29:24.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T07:29:24.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/5 2024-09-16T07:29:24.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-09-16T07:29:24.728 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.5 2024-09-16T07:29:24.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:29:24.729 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/5/whoami 2024-09-16T07:29:24.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-09-16T07:29:24.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T07:29:24.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T07:29:24.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T07:29:24.765 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:29:24.765+0000 7fce2f2f1880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:29:24.769 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:29:24.769+0000 7fce2f2f1880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:29:24.771 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:29:24.769+0000 7fce2f2f1880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:29:25.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-09-16T07:29:25.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:29:25.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-16T07:29:25.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:29:25.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:29:25.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:29:25.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:29:25.184 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T07:29:25.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:29:25.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:29:25.328 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:29:25.325+0000 7fce2f2f1880 -1 Falling back to public interface 2024-09-16T07:29:25.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:29:26.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:29:26.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:29:26.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:29:26.683 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:29:26.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:29:26.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:29:27.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:29:27.880 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:29:27.877+0000 7fce2f2f1880 -1 osd.5 80 log_to_monitors true 2024-09-16T07:29:28.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:29:28.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:29:28.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:29:28.121 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:29:28.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:29:28.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:29:28.628 INFO:tasks.workunit.client.0.smithi162.stderr:osd.5 up in weight 1 up_from 85 up_thru 78 down_at 81 last_clean_interval [30,80) [v2:127.0.0.1:6842/2933915875,v1:127.0.0.1:6843/2933915875] [v2:127.0.0.1:6844/2933915875,v1:127.0.0.1:6845/2933915875] exists,up e359a44a-e8ed-412a-942a-e6133f52f94c 2024-09-16T07:29:28.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:29:28.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:29:28.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:29:28.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-16T07:29:28.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T07:29:28.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T07:29:28.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T07:29:28.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T07:29:28.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T07:29:28.630 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T07:29:28.630 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T07:29:28.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T07:29:28.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T07:29:28.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T07:29:28.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T07:29:28.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T07:29:28.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T07:29:28.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T07:29:28.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T07:29:29.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T07:29:29.797 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:29:29.797 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:29:29.797 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:29:29.797 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:29:29.797 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:29:29.797 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-16T07:29:29.797 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-16T07:29:29.797 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-09-16T07:29:29.797 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-16T07:29:29.797 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-16T07:29:29.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T07:29:29.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:29.798 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T07:29:29.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836580 2024-09-16T07:29:29.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836580 2024-09-16T07:29:29.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836580' 2024-09-16T07:29:29.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:29.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T07:29:30.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673054 2024-09-16T07:29:30.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673054 2024-09-16T07:29:30.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836580 1-42949673054' 2024-09-16T07:29:30.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:30.179 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T07:29:30.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509529 2024-09-16T07:29:30.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509529 2024-09-16T07:29:30.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836580 1-42949673054 2-64424509529' 2024-09-16T07:29:30.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:30.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T07:29:30.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=335007449096 2024-09-16T07:29:30.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 335007449096 2024-09-16T07:29:30.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836580 1-42949673054 2-64424509529 3-335007449096' 2024-09-16T07:29:30.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:30.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T07:29:30.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182479 2024-09-16T07:29:30.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182479 2024-09-16T07:29:30.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836580 1-42949673054 2-64424509529 3-335007449096 4-107374182479' 2024-09-16T07:29:30.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:30.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T07:29:30.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=365072220163 2024-09-16T07:29:30.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 365072220163 2024-09-16T07:29:30.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836580 1-42949673054 2-64424509529 3-335007449096 4-107374182479 5-365072220163' 2024-09-16T07:29:30.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:30.995 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-16T07:29:31.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=150323855426 2024-09-16T07:29:31.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 150323855426 2024-09-16T07:29:31.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836580 1-42949673054 2-64424509529 3-335007449096 4-107374182479 5-365072220163 6-150323855426' 2024-09-16T07:29:31.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:31.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-16T07:29:31.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=171798691896 2024-09-16T07:29:31.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 171798691896 2024-09-16T07:29:31.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836580 1-42949673054 2-64424509529 3-335007449096 4-107374182479 5-365072220163 6-150323855426 7-171798691896' 2024-09-16T07:29:31.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:31.302 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-16T07:29:31.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=193273528367 2024-09-16T07:29:31.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 193273528367 2024-09-16T07:29:31.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836580 1-42949673054 2-64424509529 3-335007449096 4-107374182479 5-365072220163 6-150323855426 7-171798691896 8-193273528367' 2024-09-16T07:29:31.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:31.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-16T07:29:31.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=214748364842 2024-09-16T07:29:31.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 214748364842 2024-09-16T07:29:31.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836580 1-42949673054 2-64424509529 3-335007449096 4-107374182479 5-365072220163 6-150323855426 7-171798691896 8-193273528367 9-214748364842' 2024-09-16T07:29:31.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:31.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-16T07:29:31.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=236223201315 2024-09-16T07:29:31.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 236223201315 2024-09-16T07:29:31.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836580 1-42949673054 2-64424509529 3-335007449096 4-107374182479 5-365072220163 6-150323855426 7-171798691896 8-193273528367 9-214748364842 10-236223201315' 2024-09-16T07:29:31.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:31.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836580 2024-09-16T07:29:31.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:31.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T07:29:31.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836580 2024-09-16T07:29:31.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:31.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836580 2024-09-16T07:29:31.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836580' 2024-09-16T07:29:31.761 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836580 2024-09-16T07:29:31.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:29:32.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836581 -lt 21474836580 2024-09-16T07:29:32.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:32.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673054 2024-09-16T07:29:32.226 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:32.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T07:29:32.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673054 2024-09-16T07:29:32.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:32.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673054 2024-09-16T07:29:32.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673054' 2024-09-16T07:29:32.232 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949673054 2024-09-16T07:29:32.232 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T07:29:32.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673055 -lt 42949673054 2024-09-16T07:29:32.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:32.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509529 2024-09-16T07:29:32.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:32.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T07:29:32.755 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509529 2024-09-16T07:29:32.755 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:32.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509529 2024-09-16T07:29:32.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509529' 2024-09-16T07:29:32.756 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509529 2024-09-16T07:29:32.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T07:29:33.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509530 -lt 64424509529 2024-09-16T07:29:33.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:33.196 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-335007449096 2024-09-16T07:29:33.196 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:33.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T07:29:33.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-335007449096 2024-09-16T07:29:33.200 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:33.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=335007449096 2024-09-16T07:29:33.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 335007449096' 2024-09-16T07:29:33.201 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 335007449096 2024-09-16T07:29:33.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T07:29:33.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 335007449097 -lt 335007449096 2024-09-16T07:29:33.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:33.593 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182479 2024-09-16T07:29:33.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:33.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T07:29:33.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182479 2024-09-16T07:29:33.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:33.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182479 2024-09-16T07:29:33.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182479' 2024-09-16T07:29:33.599 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.4 seq 107374182479 2024-09-16T07:29:33.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T07:29:34.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182480 -lt 107374182479 2024-09-16T07:29:34.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:34.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-365072220163 2024-09-16T07:29:34.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:34.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T07:29:34.036 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-365072220163 2024-09-16T07:29:34.036 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:34.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=365072220163 2024-09-16T07:29:34.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 365072220163' 2024-09-16T07:29:34.038 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.5 seq 365072220163 2024-09-16T07:29:34.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T07:29:34.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 365072220164 -lt 365072220163 2024-09-16T07:29:34.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:34.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 6-150323855426 2024-09-16T07:29:34.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:34.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=6 2024-09-16T07:29:34.510 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 6-150323855426 2024-09-16T07:29:34.510 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:34.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=150323855426 2024-09-16T07:29:34.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.6 seq 150323855426' 2024-09-16T07:29:34.512 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.6 seq 150323855426 2024-09-16T07:29:34.512 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-16T07:29:34.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 150323855426 -lt 150323855426 2024-09-16T07:29:34.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:34.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 7-171798691896 2024-09-16T07:29:34.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:34.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=7 2024-09-16T07:29:34.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 7-171798691896 2024-09-16T07:29:34.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:34.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=171798691896 2024-09-16T07:29:34.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.7 seq 171798691896' 2024-09-16T07:29:34.946 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.7 seq 171798691896 2024-09-16T07:29:34.946 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-16T07:29:35.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 171798691896 -lt 171798691896 2024-09-16T07:29:35.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:35.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 8-193273528367 2024-09-16T07:29:35.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:35.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=8 2024-09-16T07:29:35.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 8-193273528367 2024-09-16T07:29:35.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:35.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=193273528367 2024-09-16T07:29:35.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.8 seq 193273528367' 2024-09-16T07:29:35.370 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.8 seq 193273528367 2024-09-16T07:29:35.370 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-16T07:29:35.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 193273528368 -lt 193273528367 2024-09-16T07:29:35.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:35.809 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 9-214748364842 2024-09-16T07:29:35.809 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:35.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=9 2024-09-16T07:29:35.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 9-214748364842 2024-09-16T07:29:35.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:35.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=214748364842 2024-09-16T07:29:35.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.9 seq 214748364842' 2024-09-16T07:29:35.814 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.9 seq 214748364842 2024-09-16T07:29:35.814 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-16T07:29:36.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 214748364843 -lt 214748364842 2024-09-16T07:29:36.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:36.204 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 10-236223201315 2024-09-16T07:29:36.204 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:36.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=10 2024-09-16T07:29:36.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 10-236223201315 2024-09-16T07:29:36.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:36.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=236223201315 2024-09-16T07:29:36.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.10 seq 236223201315' 2024-09-16T07:29:36.209 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.10 seq 236223201315 2024-09-16T07:29:36.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-16T07:29:36.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 236223201316 -lt 236223201315 2024-09-16T07:29:36.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T07:29:36.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:29:36.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:29:37.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-16T07:29:37.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T07:29:37.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T07:29:37.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T07:29:37.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T07:29:37.171 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T07:29:37.171 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T07:29:37.171 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T07:29:37.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-16T07:29:37.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T07:29:37.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:29:37.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:29:38.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-16T07:29:38.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T07:29:38.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T07:29:38.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:313: TEST_chunk_mapping: ceph osd erasure-code-profile set remap-profile plugin=lrc 'layers=[ [ "cDD", "" ] ]' mapping=_DD 'crush-steps=[ [ "choose", "osd", 0 ] ]' 2024-09-16T07:29:38.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:318: TEST_chunk_mapping: ceph osd erasure-code-profile get remap-profile 2024-09-16T07:29:39.223 INFO:tasks.workunit.client.0.smithi162.stdout:crush-device-class= 2024-09-16T07:29:39.223 INFO:tasks.workunit.client.0.smithi162.stdout:crush-failure-domain=host 2024-09-16T07:29:39.223 INFO:tasks.workunit.client.0.smithi162.stdout:crush-root=default 2024-09-16T07:29:39.223 INFO:tasks.workunit.client.0.smithi162.stdout:crush-steps=[ [ "choose", "osd", 0 ] ] 2024-09-16T07:29:39.224 INFO:tasks.workunit.client.0.smithi162.stdout:k=-1 2024-09-16T07:29:39.224 INFO:tasks.workunit.client.0.smithi162.stdout:l=-1 2024-09-16T07:29:39.224 INFO:tasks.workunit.client.0.smithi162.stdout:layers=[ [ "cDD", "" ] ] 2024-09-16T07:29:39.224 INFO:tasks.workunit.client.0.smithi162.stdout:m=-1 2024-09-16T07:29:39.224 INFO:tasks.workunit.client.0.smithi162.stdout:mapping=_DD 2024-09-16T07:29:39.224 INFO:tasks.workunit.client.0.smithi162.stdout:plugin=lrc 2024-09-16T07:29:39.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:319: TEST_chunk_mapping: create_pool remap-pool 12 12 erasure remap-profile 2024-09-16T07:29:39.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create remap-pool 12 12 erasure remap-profile 2024-09-16T07:29:40.237 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'remap-pool' already exists 2024-09-16T07:29:40.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T07:29:41.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:327: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code remap-pool 1 2 2024-09-16T07:29:41.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:281: verify_chunk_mapping: local dir=td/test-erasure-code 2024-09-16T07:29:41.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: local poolname=remap-pool 2024-09-16T07:29:41.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:283: verify_chunk_mapping: local first=1 2024-09-16T07:29:41.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:284: verify_chunk_mapping: local second=2 2024-09-16T07:29:41.253 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-09-16T07:29:41.253 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-16T07:29:41.278 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-09-16T07:29:41.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 FIRSTremap-pool 2024-09-16T07:29:41.279 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-09-16T07:29:41.280 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-16T07:29:41.300 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-09-16T07:29:41.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 SECONDremap-pool 2024-09-16T07:29:41.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: local 'payload= FIRSTremap-pool SECONDremap-pool' 2024-09-16T07:29:41.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: echo -n ' FIRSTremap-pool SECONDremap-pool' 2024-09-16T07:29:41.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: rados --pool remap-pool put SOMETHINGremap-pool td/test-erasure-code/ORIGINAL 2024-09-16T07:29:42.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:290: verify_chunk_mapping: rados --pool remap-pool get SOMETHINGremap-pool td/test-erasure-code/COPY 2024-09-16T07:29:43.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: osds=($(get_osds $poolname SOMETHING$poolname)) 2024-09-16T07:29:43.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2024-09-16T07:29:43.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=remap-pool 2024-09-16T07:29:43.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGremap-pool 2024-09-16T07:29:43.178 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2024-09-16T07:29:43.178 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:29:43.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-09-16T07:29:43.649 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-16T07:29:43.649 INFO:tasks.workunit.client.0.smithi162.stderr:9' 2024-09-16T07:29:43.649 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 6 9 2024-09-16T07:29:43.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: local -a osds 2024-09-16T07:29:43.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i = 0 )) 2024-09-16T07:29:43.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-09-16T07:29:43.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.4 flush_journal 2024-09-16T07:29:43.772 INFO:tasks.workunit.client.0.smithi162.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-16T07:29:43.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-16T07:29:43.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-09-16T07:29:43.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.6 flush_journal 2024-09-16T07:29:43.903 INFO:tasks.workunit.client.0.smithi162.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-16T07:29:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-16T07:29:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-09-16T07:29:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2024-09-16T07:29:44.027 INFO:tasks.workunit.client.0.smithi162.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-16T07:29:44.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-16T07:29:44.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-09-16T07:29:44.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-16T07:29:44.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: rm td/test-erasure-code/COPY 2024-09-16T07:29:44.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: osds=($(get_osds $poolname SOMETHING$poolname)) 2024-09-16T07:29:44.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2024-09-16T07:29:44.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=remap-pool 2024-09-16T07:29:44.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGremap-pool 2024-09-16T07:29:44.033 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2024-09-16T07:29:44.034 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:29:44.504 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-09-16T07:29:44.504 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-16T07:29:44.504 INFO:tasks.workunit.client.0.smithi162.stderr:9' 2024-09-16T07:29:44.505 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 6 9 2024-09-16T07:29:44.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: local -a osds 2024-09-16T07:29:44.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: grep --quiet FIRSTremap-pool 2024-09-16T07:29:44.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: objectstore_tool td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-09-16T07:29:44.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-09-16T07:29:44.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T07:29:44.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=6 2024-09-16T07:29:44.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T07:29:44.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-09-16T07:29:44.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-09-16T07:29:44.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T07:29:44.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=6 2024-09-16T07:29:44.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T07:29:44.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.6 2024-09-16T07:29:44.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:29:44.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:29:44.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:29:44.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:29:44.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:29:45.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:29:45.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-09-16T07:29:45.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-09-16T07:29:45.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T07:29:45.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=6 2024-09-16T07:29:45.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T07:29:45.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/6 2024-09-16T07:29:45.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/6 SOMETHINGremap-pool get-bytes 2024-09-16T07:29:48.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 6 2024-09-16T07:29:48.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-09-16T07:29:48.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T07:29:48.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=6 2024-09-16T07:29:48.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T07:29:48.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/6 2024-09-16T07:29:48.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-16T07:29:48.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:29:48.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:29:48.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:29:48.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2024-09-16T07:29:48.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2024-09-16T07:29:48.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T07:29:48.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T07:29:48.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-16T07:29:48.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T07:29:48.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:29:48.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:29:48.186 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:29:48.186 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:29:48.186 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:29:48.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:29:48.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:29:48.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:29:48.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-16T07:29:48.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-16T07:29:48.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:29:48.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:29:48.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:29:48.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:29:48.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T07:29:48.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T07:29:48.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/6 2024-09-16T07:29:48.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.6 2024-09-16T07:29:48.188 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.6 2024-09-16T07:29:48.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 6 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:29:48.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/6/whoami 2024-09-16T07:29:48.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 6 = 6 ']' 2024-09-16T07:29:48.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T07:29:48.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T07:29:48.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T07:29:48.224 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:29:48.221+0000 7f9bf54f0880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:29:48.227 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:29:48.225+0000 7f9bf54f0880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:29:48.229 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:29:48.225+0000 7f9bf54f0880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:29:48.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 6 2024-09-16T07:29:48.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:29:48.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-09-16T07:29:48.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:29:48.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:29:48.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:29:48.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:29:48.661 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T07:29:48.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:29:48.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:29:48.795 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:29:48.793+0000 7f9bf54f0880 -1 Falling back to public interface 2024-09-16T07:29:49.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:29:50.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:29:50.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:29:50.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:29:50.188 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:29:50.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:29:50.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:29:50.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:29:51.363 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:29:51.361+0000 7f9bf54f0880 -1 osd.6 94 log_to_monitors true 2024-09-16T07:29:51.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:29:51.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:29:51.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:29:51.661 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:29:51.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:29:51.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:29:52.155 INFO:tasks.workunit.client.0.smithi162.stderr:osd.6 up in weight 1 up_from 99 up_thru 90 down_at 95 last_clean_interval [35,94) [v2:127.0.0.1:6850/1494171362,v1:127.0.0.1:6851/1494171362] [v2:127.0.0.1:6852/1494171362,v1:127.0.0.1:6853/1494171362] exists,up 377c6e19-55cf-4eee-a1da-650f297ae4c6 2024-09-16T07:29:52.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:29:52.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:29:52.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:29:52.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-16T07:29:52.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T07:29:52.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T07:29:52.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T07:29:52.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T07:29:52.157 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T07:29:52.158 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T07:29:52.158 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T07:29:52.160 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T07:29:52.160 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T07:29:52.160 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T07:29:52.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T07:29:52.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T07:29:52.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T07:29:52.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T07:29:52.348 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T07:29:53.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T07:29:53.234 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:29:53.234 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:29:53.234 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:29:53.234 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:29:53.234 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:29:53.234 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-16T07:29:53.234 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-16T07:29:53.234 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-09-16T07:29:53.234 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-16T07:29:53.235 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-16T07:29:53.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T07:29:53.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:53.235 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T07:29:53.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836587 2024-09-16T07:29:53.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836587 2024-09-16T07:29:53.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836587' 2024-09-16T07:29:53.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:53.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T07:29:53.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673061 2024-09-16T07:29:53.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673061 2024-09-16T07:29:53.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836587 1-42949673061' 2024-09-16T07:29:53.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:53.987 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T07:29:54.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509536 2024-09-16T07:29:54.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509536 2024-09-16T07:29:54.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836587 1-42949673061 2-64424509536' 2024-09-16T07:29:54.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:54.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T07:29:54.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=335007449103 2024-09-16T07:29:54.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 335007449103 2024-09-16T07:29:54.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836587 1-42949673061 2-64424509536 3-335007449103' 2024-09-16T07:29:54.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:54.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T07:29:54.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182486 2024-09-16T07:29:54.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182486 2024-09-16T07:29:54.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836587 1-42949673061 2-64424509536 3-335007449103 4-107374182486' 2024-09-16T07:29:54.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:54.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T07:29:55.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=365072220170 2024-09-16T07:29:55.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 365072220170 2024-09-16T07:29:55.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836587 1-42949673061 2-64424509536 3-335007449103 4-107374182486 5-365072220170' 2024-09-16T07:29:55.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:55.002 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-16T07:29:55.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=425201762307 2024-09-16T07:29:55.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 425201762307 2024-09-16T07:29:55.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836587 1-42949673061 2-64424509536 3-335007449103 4-107374182486 5-365072220170 6-425201762307' 2024-09-16T07:29:55.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:55.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-16T07:29:55.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=171798691903 2024-09-16T07:29:55.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 171798691903 2024-09-16T07:29:55.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836587 1-42949673061 2-64424509536 3-335007449103 4-107374182486 5-365072220170 6-425201762307 7-171798691903' 2024-09-16T07:29:55.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:55.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-16T07:29:55.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=193273528374 2024-09-16T07:29:55.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 193273528374 2024-09-16T07:29:55.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836587 1-42949673061 2-64424509536 3-335007449103 4-107374182486 5-365072220170 6-425201762307 7-171798691903 8-193273528374' 2024-09-16T07:29:55.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:55.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-16T07:29:55.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=214748364849 2024-09-16T07:29:55.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 214748364849 2024-09-16T07:29:55.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836587 1-42949673061 2-64424509536 3-335007449103 4-107374182486 5-365072220170 6-425201762307 7-171798691903 8-193273528374 9-214748364849' 2024-09-16T07:29:55.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:29:55.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-16T07:29:55.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=236223201322 2024-09-16T07:29:55.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 236223201322 2024-09-16T07:29:55.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836587 1-42949673061 2-64424509536 3-335007449103 4-107374182486 5-365072220170 6-425201762307 7-171798691903 8-193273528374 9-214748364849 10-236223201322' 2024-09-16T07:29:55.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:55.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:55.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836587 2024-09-16T07:29:55.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T07:29:55.764 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836587 2024-09-16T07:29:55.764 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:55.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836587 2024-09-16T07:29:55.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836587' 2024-09-16T07:29:55.766 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836587 2024-09-16T07:29:55.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:29:56.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836587 -lt 21474836587 2024-09-16T07:29:56.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:56.332 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673061 2024-09-16T07:29:56.332 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:56.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T07:29:56.335 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673061 2024-09-16T07:29:56.335 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:56.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673061 2024-09-16T07:29:56.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673061' 2024-09-16T07:29:56.337 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949673061 2024-09-16T07:29:56.338 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T07:29:56.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673061 -lt 42949673061 2024-09-16T07:29:56.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:56.817 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509536 2024-09-16T07:29:56.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:56.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T07:29:56.821 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509536 2024-09-16T07:29:56.821 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:56.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509536 2024-09-16T07:29:56.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2024-09-16T07:29:56.823 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509536 2024-09-16T07:29:56.824 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T07:29:57.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509536 -lt 64424509536 2024-09-16T07:29:57.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:57.309 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:57.309 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-335007449103 2024-09-16T07:29:57.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T07:29:57.311 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-335007449103 2024-09-16T07:29:57.311 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:57.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=335007449103 2024-09-16T07:29:57.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 335007449103' 2024-09-16T07:29:57.314 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 335007449103 2024-09-16T07:29:57.314 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T07:29:57.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 335007449104 -lt 335007449103 2024-09-16T07:29:57.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:57.771 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182486 2024-09-16T07:29:57.771 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:57.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T07:29:57.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182486 2024-09-16T07:29:57.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:57.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182486 2024-09-16T07:29:57.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182486' 2024-09-16T07:29:57.775 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.4 seq 107374182486 2024-09-16T07:29:57.776 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T07:29:58.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182487 -lt 107374182486 2024-09-16T07:29:58.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:58.232 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-365072220170 2024-09-16T07:29:58.232 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:58.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T07:29:58.235 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-365072220170 2024-09-16T07:29:58.235 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:58.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=365072220170 2024-09-16T07:29:58.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 365072220170' 2024-09-16T07:29:58.237 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.5 seq 365072220170 2024-09-16T07:29:58.237 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T07:29:58.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 365072220170 -lt 365072220170 2024-09-16T07:29:58.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:58.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 6-425201762307 2024-09-16T07:29:58.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:58.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=6 2024-09-16T07:29:58.703 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 6-425201762307 2024-09-16T07:29:58.703 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:58.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=425201762307 2024-09-16T07:29:58.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.6 seq 425201762307' 2024-09-16T07:29:58.706 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.6 seq 425201762307 2024-09-16T07:29:58.706 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-16T07:29:59.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 425201762308 -lt 425201762307 2024-09-16T07:29:59.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:59.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 7-171798691903 2024-09-16T07:29:59.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:59.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=7 2024-09-16T07:29:59.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 7-171798691903 2024-09-16T07:29:59.188 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:59.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=171798691903 2024-09-16T07:29:59.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.7 seq 171798691903' 2024-09-16T07:29:59.191 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.7 seq 171798691903 2024-09-16T07:29:59.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-16T07:29:59.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 171798691903 -lt 171798691903 2024-09-16T07:29:59.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:29:59.682 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 8-193273528374 2024-09-16T07:29:59.682 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:29:59.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=8 2024-09-16T07:29:59.685 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 8-193273528374 2024-09-16T07:29:59.685 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:29:59.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=193273528374 2024-09-16T07:29:59.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.8 seq 193273528374' 2024-09-16T07:29:59.687 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.8 seq 193273528374 2024-09-16T07:29:59.687 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-16T07:30:00.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 193273528374 -lt 193273528374 2024-09-16T07:30:00.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:30:00.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:30:00.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 9-214748364849 2024-09-16T07:30:00.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=9 2024-09-16T07:30:00.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 9-214748364849 2024-09-16T07:30:00.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:30:00.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=214748364849 2024-09-16T07:30:00.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.9 seq 214748364849' 2024-09-16T07:30:00.194 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.9 seq 214748364849 2024-09-16T07:30:00.194 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-16T07:30:00.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 214748364850 -lt 214748364849 2024-09-16T07:30:00.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:30:00.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 10-236223201322 2024-09-16T07:30:00.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:30:00.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=10 2024-09-16T07:30:00.646 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:30:00.646 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 10-236223201322 2024-09-16T07:30:00.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=236223201322 2024-09-16T07:30:00.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.10 seq 236223201322' 2024-09-16T07:30:00.647 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.10 seq 236223201322 2024-09-16T07:30:00.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-16T07:30:01.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 236223201323 -lt 236223201322 2024-09-16T07:30:01.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T07:30:01.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:30:01.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:30:01.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-09-16T07:30:01.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T07:30:01.713 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T07:30:01.713 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T07:30:01.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T07:30:01.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T07:30:01.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T07:30:01.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T07:30:02.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-09-16T07:30:02.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T07:30:02.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:30:02.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:30:02.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-09-16T07:30:02.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T07:30:02.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T07:30:02.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: objectstore_tool td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-09-16T07:30:02.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-09-16T07:30:02.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T07:30:02.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-09-16T07:30:02.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T07:30:02.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-09-16T07:30:02.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-09-16T07:30:02.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T07:30:02.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-09-16T07:30:02.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T07:30:02.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.9 2024-09-16T07:30:02.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: grep --quiet SECONDremap-pool 2024-09-16T07:30:02.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:30:02.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:30:02.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:30:02.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:30:02.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:30:03.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:30:03.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-09-16T07:30:03.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-09-16T07:30:03.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T07:30:03.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-09-16T07:30:03.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T07:30:03.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/9 2024-09-16T07:30:03.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/9 SOMETHINGremap-pool get-bytes 2024-09-16T07:30:06.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 9 2024-09-16T07:30:06.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-09-16T07:30:06.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T07:30:06.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-09-16T07:30:06.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T07:30:06.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/9 2024-09-16T07:30:06.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-16T07:30:06.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:30:06.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:30:06.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:30:06.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2024-09-16T07:30:06.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2024-09-16T07:30:06.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T07:30:06.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T07:30:06.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-16T07:30:06.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T07:30:06.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:30:06.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:30:06.145 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:30:06.146 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:30:06.146 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:30:06.146 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:30:06.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' 2024-09-16T07:30:06.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:30:06.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-16T07:30:06.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-16T07:30:06.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:30:06.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:30:06.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:30:06.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:30:06.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T07:30:06.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T07:30:06.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/9 2024-09-16T07:30:06.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-09-16T07:30:06.147 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.9 2024-09-16T07:30:06.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=85cfa5b8-839e-46b0-9938-11ef39674051 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46203/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:30:06.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/9/whoami 2024-09-16T07:30:06.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-09-16T07:30:06.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T07:30:06.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T07:30:06.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T07:30:06.183 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:30:06.180+0000 7f0bfa74a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:30:06.186 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:30:06.184+0000 7f0bfa74a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:30:06.188 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:30:06.184+0000 7f0bfa74a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:30:06.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-09-16T07:30:06.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:30:06.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-09-16T07:30:06.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:30:06.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:30:06.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:30:06.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:30:06.581 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T07:30:06.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:30:06.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-16T07:30:06.744 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:30:06.740+0000 7f0bfa74a880 -1 Falling back to public interface 2024-09-16T07:30:07.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:30:08.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:30:08.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:30:08.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:30:08.057 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:30:08.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:30:08.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-16T07:30:08.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:30:09.403 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:30:09.400+0000 7f0bfa74a880 -1 osd.9 102 log_to_monitors true 2024-09-16T07:30:09.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:30:09.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:30:09.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:30:09.537 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:30:09.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:30:09.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-16T07:30:10.321 INFO:tasks.workunit.client.0.smithi162.stderr:osd.9 up in weight 1 up_from 107 up_thru 68 down_at 103 last_clean_interval [50,102) [v2:127.0.0.1:6874/867221456,v1:127.0.0.1:6875/867221456] [v2:127.0.0.1:6876/867221456,v1:127.0.0.1:6877/867221456] exists,up 1219e850-bad2-4898-8efe-09fa3f2b1315 2024-09-16T07:30:10.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:30:10.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:30:10.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:30:10.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-16T07:30:10.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T07:30:10.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T07:30:10.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T07:30:10.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T07:30:10.322 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T07:30:10.322 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T07:30:10.322 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T07:30:10.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T07:30:10.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T07:30:10.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T07:30:10.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T07:30:10.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T07:30:10.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T07:30:10.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T07:30:10.515 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T07:30:11.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T07:30:11.455 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:30:11.455 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:30:11.456 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:30:11.456 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:30:11.456 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:30:11.456 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-16T07:30:11.456 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-16T07:30:11.456 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-09-16T07:30:11.456 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-16T07:30:11.456 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-16T07:30:11.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T07:30:11.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:30:11.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T07:30:11.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836593 2024-09-16T07:30:11.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836593 2024-09-16T07:30:11.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836593' 2024-09-16T07:30:11.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:30:11.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T07:30:12.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673067 2024-09-16T07:30:12.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673067 2024-09-16T07:30:12.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836593 1-42949673067' 2024-09-16T07:30:12.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:30:12.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T07:30:12.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509542 2024-09-16T07:30:12.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509542 2024-09-16T07:30:12.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836593 1-42949673067 2-64424509542' 2024-09-16T07:30:12.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:30:12.451 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T07:30:12.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=335007449109 2024-09-16T07:30:12.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 335007449109 2024-09-16T07:30:12.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836593 1-42949673067 2-64424509542 3-335007449109' 2024-09-16T07:30:12.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:30:12.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T07:30:13.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182492 2024-09-16T07:30:13.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182492 2024-09-16T07:30:13.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836593 1-42949673067 2-64424509542 3-335007449109 4-107374182492' 2024-09-16T07:30:13.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:30:13.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T07:30:13.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=365072220175 2024-09-16T07:30:13.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 365072220175 2024-09-16T07:30:13.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836593 1-42949673067 2-64424509542 3-335007449109 4-107374182492 5-365072220175' 2024-09-16T07:30:13.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:30:13.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-16T07:30:13.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=425201762313 2024-09-16T07:30:13.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 425201762313 2024-09-16T07:30:13.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836593 1-42949673067 2-64424509542 3-335007449109 4-107374182492 5-365072220175 6-425201762313' 2024-09-16T07:30:13.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:30:13.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-16T07:30:13.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=171798691908 2024-09-16T07:30:13.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 171798691908 2024-09-16T07:30:13.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836593 1-42949673067 2-64424509542 3-335007449109 4-107374182492 5-365072220175 6-425201762313 7-171798691908' 2024-09-16T07:30:13.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:30:13.518 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-16T07:30:13.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=193273528379 2024-09-16T07:30:13.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 193273528379 2024-09-16T07:30:13.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836593 1-42949673067 2-64424509542 3-335007449109 4-107374182492 5-365072220175 6-425201762313 7-171798691908 8-193273528379' 2024-09-16T07:30:13.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:30:13.670 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-16T07:30:13.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=459561500675 2024-09-16T07:30:13.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 459561500675 2024-09-16T07:30:13.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836593 1-42949673067 2-64424509542 3-335007449109 4-107374182492 5-365072220175 6-425201762313 7-171798691908 8-193273528379 9-459561500675' 2024-09-16T07:30:13.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:30:13.827 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-16T07:30:13.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=236223201327 2024-09-16T07:30:13.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 236223201327 2024-09-16T07:30:13.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836593 1-42949673067 2-64424509542 3-335007449109 4-107374182492 5-365072220175 6-425201762313 7-171798691908 8-193273528379 9-459561500675 10-236223201327' 2024-09-16T07:30:13.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:30:13.977 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:30:13.978 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836593 2024-09-16T07:30:13.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T07:30:13.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836593 2024-09-16T07:30:13.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:30:13.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836593 2024-09-16T07:30:13.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836593' 2024-09-16T07:30:13.981 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836593 2024-09-16T07:30:13.981 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:30:14.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836593 -lt 21474836593 2024-09-16T07:30:14.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:30:14.436 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673067 2024-09-16T07:30:14.436 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:30:14.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T07:30:14.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673067 2024-09-16T07:30:14.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:30:14.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673067 2024-09-16T07:30:14.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673067' 2024-09-16T07:30:14.441 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949673067 2024-09-16T07:30:14.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T07:30:14.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673067 -lt 42949673067 2024-09-16T07:30:14.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:30:14.951 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509542 2024-09-16T07:30:14.951 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:30:14.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T07:30:14.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509542 2024-09-16T07:30:14.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:30:14.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509542 2024-09-16T07:30:14.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509542' 2024-09-16T07:30:14.956 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509542 2024-09-16T07:30:14.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T07:30:15.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509542 -lt 64424509542 2024-09-16T07:30:15.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:30:15.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-335007449109 2024-09-16T07:30:15.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:30:15.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T07:30:15.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-335007449109 2024-09-16T07:30:15.406 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:30:15.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=335007449109 2024-09-16T07:30:15.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 335007449109' 2024-09-16T07:30:15.408 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 335007449109 2024-09-16T07:30:15.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T07:30:15.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 335007449109 -lt 335007449109 2024-09-16T07:30:15.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:30:15.860 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182492 2024-09-16T07:30:15.860 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:30:15.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T07:30:15.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:30:15.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182492 2024-09-16T07:30:15.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182492 2024-09-16T07:30:15.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182492' 2024-09-16T07:30:15.865 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.4 seq 107374182492 2024-09-16T07:30:15.865 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T07:30:16.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182492 -lt 107374182492 2024-09-16T07:30:16.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:30:16.326 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-365072220175 2024-09-16T07:30:16.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:30:16.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T07:30:16.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-365072220175 2024-09-16T07:30:16.330 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:30:16.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=365072220175 2024-09-16T07:30:16.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 365072220175' 2024-09-16T07:30:16.332 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.5 seq 365072220175 2024-09-16T07:30:16.332 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T07:30:16.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 365072220176 -lt 365072220175 2024-09-16T07:30:16.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:30:16.868 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 6-425201762313 2024-09-16T07:30:16.868 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:30:16.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=6 2024-09-16T07:30:16.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 6-425201762313 2024-09-16T07:30:16.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:30:16.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=425201762313 2024-09-16T07:30:16.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.6 seq 425201762313' 2024-09-16T07:30:16.873 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.6 seq 425201762313 2024-09-16T07:30:16.873 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-16T07:30:17.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 425201762313 -lt 425201762313 2024-09-16T07:30:17.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:30:17.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 7-171798691908 2024-09-16T07:30:17.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:30:17.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=7 2024-09-16T07:30:17.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 7-171798691908 2024-09-16T07:30:17.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:30:17.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=171798691908 2024-09-16T07:30:17.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.7 seq 171798691908' 2024-09-16T07:30:17.305 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.7 seq 171798691908 2024-09-16T07:30:17.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-16T07:30:17.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 171798691909 -lt 171798691908 2024-09-16T07:30:17.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:30:17.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 8-193273528379 2024-09-16T07:30:17.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:30:17.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=8 2024-09-16T07:30:17.730 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:30:17.730 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 8-193273528379 2024-09-16T07:30:17.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=193273528379 2024-09-16T07:30:17.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.8 seq 193273528379' 2024-09-16T07:30:17.731 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.8 seq 193273528379 2024-09-16T07:30:17.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-16T07:30:18.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 193273528380 -lt 193273528379 2024-09-16T07:30:18.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:30:18.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 9-459561500675 2024-09-16T07:30:18.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:30:18.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=9 2024-09-16T07:30:18.166 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 9-459561500675 2024-09-16T07:30:18.166 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:30:18.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=459561500675 2024-09-16T07:30:18.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.9 seq 459561500675' 2024-09-16T07:30:18.168 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.9 seq 459561500675 2024-09-16T07:30:18.168 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-16T07:30:18.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 459561500676 -lt 459561500675 2024-09-16T07:30:18.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:30:18.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 10-236223201327 2024-09-16T07:30:18.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:30:18.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=10 2024-09-16T07:30:18.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 10-236223201327 2024-09-16T07:30:18.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:30:18.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=236223201327 2024-09-16T07:30:18.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.10 seq 236223201327' 2024-09-16T07:30:18.653 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.10 seq 236223201327 2024-09-16T07:30:18.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-16T07:30:19.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 236223201328 -lt 236223201327 2024-09-16T07:30:19.112 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T07:30:19.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:30:19.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:30:19.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-09-16T07:30:19.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T07:30:19.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T07:30:19.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T07:30:19.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T07:30:19.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T07:30:19.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T07:30:19.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T07:30:20.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-09-16T07:30:20.103 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T07:30:20.103 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:30:20.104 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:30:20.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-09-16T07:30:20.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T07:30:20.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T07:30:20.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:329: TEST_chunk_mapping: delete_pool remap-pool 2024-09-16T07:30:20.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=remap-pool 2024-09-16T07:30:20.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete remap-pool remap-pool --yes-i-really-really-mean-it 2024-09-16T07:30:21.263 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'remap-pool' does not exist 2024-09-16T07:30:21.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:330: TEST_chunk_mapping: ceph osd erasure-code-profile rm remap-profile 2024-09-16T07:30:21.960 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile remap-profile does not exist 2024-09-16T07:30:21.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-16T07:30:21.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_isa td/test-erasure-code 2024-09-16T07:30:21.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:193: TEST_rados_put_get_isa: erasure_code_plugin_exists isa 2024-09-16T07:30:21.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: erasure_code_plugin_exists: local plugin=isa 2024-09-16T07:30:21.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2007: erasure_code_plugin_exists: local status 2024-09-16T07:30:21.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2008: erasure_code_plugin_exists: local grepstr 2024-09-16T07:30:21.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: erasure_code_plugin_exists: local s 2024-09-16T07:30:21.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: erasure_code_plugin_exists: case `uname` in 2024-09-16T07:30:21.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: erasure_code_plugin_exists: uname 2024-09-16T07:30:21.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2012: erasure_code_plugin_exists: grepstr='isa.*No such file' 2024-09-16T07:30:21.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=isa 2024-09-16T07:30:23.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: erasure_code_plugin_exists: s= 2024-09-16T07:30:23.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: erasure_code_plugin_exists: local status=0 2024-09-16T07:30:23.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2024-09-16T07:30:23.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2024-09-16T07:30:23.955 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile TESTPROFILE does not exist 2024-09-16T07:30:23.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2024: erasure_code_plugin_exists: return 0 2024-09-16T07:30:23.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:197: TEST_rados_put_get_isa: local dir=td/test-erasure-code 2024-09-16T07:30:23.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:198: TEST_rados_put_get_isa: local poolname=pool-isa 2024-09-16T07:30:23.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:200: TEST_rados_put_get_isa: ceph osd erasure-code-profile set profile-isa plugin=isa crush-failure-domain=osd 2024-09-16T07:30:24.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:203: TEST_rados_put_get_isa: create_pool pool-isa 1 1 erasure profile-isa 2024-09-16T07:30:24.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-isa 1 1 erasure profile-isa 2024-09-16T07:30:25.597 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-isa' already exists 2024-09-16T07:30:25.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T07:30:26.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:206: TEST_rados_put_get_isa: rados_put_get td/test-erasure-code pool-isa 2024-09-16T07:30:26.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-16T07:30:26.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-isa 2024-09-16T07:30:26.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-16T07:30:26.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-16T07:30:26.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-16T07:30:26.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-16T07:30:26.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-16T07:30:26.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-16T07:30:26.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-16T07:30:26.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-16T07:30:26.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-16T07:30:26.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-isa put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-16T07:30:27.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2024-09-16T07:30:27.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-16T07:30:27.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-16T07:30:27.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T07:30:27.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-isa SOMETHING 2024-09-16T07:30:27.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-isa 2024-09-16T07:30:27.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-16T07:30:27.864 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-isa SOMETHING 2024-09-16T07:30:27.864 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:30:28.312 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-09-16T07:30:28.312 INFO:tasks.workunit.client.0.smithi162.stderr:10 2024-09-16T07:30:28.312 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T07:30:28.313 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-16T07:30:28.313 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-16T07:30:28.313 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-09-16T07:30:28.313 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:30:28.313 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:30:28.313 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:30:28.313 INFO:tasks.workunit.client.0.smithi162.stderr:9' 2024-09-16T07:30:28.313 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 10 0 7 6 8 2 1 5 9 2024-09-16T07:30:28.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-16T07:30:28.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=9 2024-09-16T07:30:28.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2024-09-16T07:30:28.898 INFO:tasks.workunit.client.0.smithi162.stderr:osd.9 is already out. 2024-09-16T07:30:28.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-09-16T07:30:33.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-isa SOMETHING 2024-09-16T07:30:33.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-isa 2024-09-16T07:30:33.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-16T07:30:33.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<9\>' 2024-09-16T07:30:33.911 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-isa SOMETHING 2024-09-16T07:30:33.911 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:30:34.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-09-16T07:30:34.358 INFO:tasks.workunit.client.0.smithi162.stderr:10 2024-09-16T07:30:34.358 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T07:30:34.358 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-16T07:30:34.358 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-16T07:30:34.358 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-09-16T07:30:34.358 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:30:34.358 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:30:34.358 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:30:34.359 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T07:30:34.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 10 0 7 6 8 3 1 5 2 2024-09-16T07:30:34.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2024-09-16T07:30:34.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-16T07:30:34.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 9 2024-09-16T07:30:35.028 INFO:tasks.workunit.client.0.smithi162.stderr:osd.9 is already in. 2024-09-16T07:30:35.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-16T07:30:35.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:208: TEST_rados_put_get_isa: delete_pool pool-isa 2024-09-16T07:30:35.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-isa 2024-09-16T07:30:35.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-isa pool-isa --yes-i-really-really-mean-it 2024-09-16T07:30:36.985 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-isa' does not exist 2024-09-16T07:30:36.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-16T07:30:36.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_jerasure td/test-erasure-code 2024-09-16T07:30:36.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:212: TEST_rados_put_get_jerasure: local dir=td/test-erasure-code 2024-09-16T07:30:36.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:214: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code ecpool 2024-09-16T07:30:36.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-16T07:30:36.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=ecpool 2024-09-16T07:30:36.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-16T07:30:36.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-16T07:30:36.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-16T07:30:37.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-16T07:30:37.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-16T07:30:37.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-16T07:30:37.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-16T07:30:37.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-16T07:30:37.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-16T07:30:37.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool ecpool put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-16T07:30:38.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2024-09-16T07:30:38.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-16T07:30:38.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-16T07:30:38.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T07:30:38.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds ecpool SOMETHING 2024-09-16T07:30:38.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-09-16T07:30:38.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-16T07:30:38.439 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-16T07:30:38.439 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:30:39.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=6 2024-09-16T07:30:39.150 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T07:30:39.150 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-16T07:30:39.150 INFO:tasks.workunit.client.0.smithi162.stderr:9' 2024-09-16T07:30:39.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 6 0 7 9 2024-09-16T07:30:39.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-16T07:30:39.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=3 2024-09-16T07:30:39.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2024-09-16T07:30:39.766 INFO:tasks.workunit.client.0.smithi162.stderr:osd.9 is already out. 2024-09-16T07:30:39.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-09-16T07:30:44.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds ecpool SOMETHING 2024-09-16T07:30:44.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-09-16T07:30:44.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-16T07:30:44.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<9\>' 2024-09-16T07:30:44.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-16T07:30:44.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:30:45.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=6 2024-09-16T07:30:45.218 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T07:30:45.218 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-16T07:30:45.218 INFO:tasks.workunit.client.0.smithi162.stderr:5' 2024-09-16T07:30:45.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 6 0 7 5 2024-09-16T07:30:45.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2024-09-16T07:30:45.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-16T07:30:45.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 9 2024-09-16T07:30:46.131 INFO:tasks.workunit.client.0.smithi162.stderr:osd.9 is already in. 2024-09-16T07:30:46.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-16T07:30:46.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:216: TEST_rados_put_get_jerasure: local poolname=pool-jerasure 2024-09-16T07:30:46.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:217: TEST_rados_put_get_jerasure: local profile=profile-jerasure 2024-09-16T07:30:46.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:219: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile set profile-jerasure plugin=jerasure k=4 m=2 crush-failure-domain=osd 2024-09-16T07:30:47.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:223: TEST_rados_put_get_jerasure: create_pool pool-jerasure 12 12 erasure profile-jerasure 2024-09-16T07:30:47.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 12 12 erasure profile-jerasure 2024-09-16T07:30:49.398 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-16T07:30:49.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T07:30:50.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:226: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code pool-jerasure 2024-09-16T07:30:50.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-16T07:30:50.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-jerasure 2024-09-16T07:30:50.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-16T07:30:50.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-16T07:30:50.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-16T07:30:50.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-16T07:30:50.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-16T07:30:50.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-16T07:30:50.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-16T07:30:50.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-16T07:30:50.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-16T07:30:50.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-16T07:30:53.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-09-16T07:30:53.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-16T07:30:53.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-16T07:30:53.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T07:30:53.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-jerasure SOMETHING 2024-09-16T07:30:53.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T07:30:53.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-16T07:30:53.355 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-16T07:30:53.355 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:30:53.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-09-16T07:30:53.994 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:30:53.994 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-16T07:30:53.994 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:30:53.994 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:30:53.994 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-16T07:30:53.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 10 2024-09-16T07:30:53.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-16T07:30:53.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=5 2024-09-16T07:30:53.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 10 2024-09-16T07:30:54.836 INFO:tasks.workunit.client.0.smithi162.stderr:osd.10 is already out. 2024-09-16T07:30:54.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-09-16T07:30:59.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-jerasure SOMETHING 2024-09-16T07:30:59.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T07:30:59.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-16T07:30:59.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<10\>' 2024-09-16T07:30:59.852 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-16T07:30:59.852 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:31:00.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-09-16T07:31:00.296 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:31:00.296 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-16T07:31:00.296 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:31:00.296 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:31:00.296 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-16T07:31:00.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 6 2024-09-16T07:31:00.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-09-16T07:31:00.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-16T07:31:00.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 10 2024-09-16T07:31:00.999 INFO:tasks.workunit.client.0.smithi162.stderr:osd.10 is already in. 2024-09-16T07:31:01.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-16T07:31:01.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:227: TEST_rados_put_get_jerasure: rados_osds_out_in td/test-erasure-code pool-jerasure 2024-09-16T07:31:01.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:104: rados_osds_out_in: local dir=td/test-erasure-code 2024-09-16T07:31:01.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:105: rados_osds_out_in: local poolname=pool-jerasure 2024-09-16T07:31:01.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:106: rados_osds_out_in: local objname=SOMETHING 2024-09-16T07:31:01.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-09-16T07:31:01.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 FFFF 2024-09-16T07:31:01.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-09-16T07:31:01.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 GGGG 2024-09-16T07:31:01.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-09-16T07:31:01.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 HHHH 2024-09-16T07:31:01.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-09-16T07:31:01.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 IIII 2024-09-16T07:31:01.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:116: rados_osds_out_in: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-16T07:31:04.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:117: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-09-16T07:31:04.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:118: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-16T07:31:04.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:119: rados_osds_out_in: rm td/test-erasure-code/COPY 2024-09-16T07:31:04.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:127: rados_osds_out_in: wait_for_clean 2024-09-16T07:31:04.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T07:31:04.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T07:31:04.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T07:31:04.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T07:31:04.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T07:31:04.767 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T07:31:04.767 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T07:31:04.767 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T07:31:04.767 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T07:31:04.767 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T07:31:04.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T07:31:04.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T07:31:04.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T07:31:04.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T07:31:04.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T07:31:05.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T07:31:05.384 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:31:05.384 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:31:05.384 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:31:05.384 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:31:05.384 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:31:05.384 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-16T07:31:05.384 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-16T07:31:05.384 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-09-16T07:31:05.385 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-16T07:31:05.385 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-16T07:31:05.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T07:31:05.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:05.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T07:31:05.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836605 2024-09-16T07:31:05.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836605 2024-09-16T07:31:05.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836605' 2024-09-16T07:31:05.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:05.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T07:31:05.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673079 2024-09-16T07:31:05.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673079 2024-09-16T07:31:05.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836605 1-42949673079' 2024-09-16T07:31:05.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:05.709 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T07:31:05.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509554 2024-09-16T07:31:05.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509554 2024-09-16T07:31:05.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836605 1-42949673079 2-64424509554' 2024-09-16T07:31:05.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:05.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T07:31:06.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=335007449121 2024-09-16T07:31:06.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 335007449121 2024-09-16T07:31:06.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836605 1-42949673079 2-64424509554 3-335007449121' 2024-09-16T07:31:06.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:06.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T07:31:06.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182504 2024-09-16T07:31:06.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182504 2024-09-16T07:31:06.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836605 1-42949673079 2-64424509554 3-335007449121 4-107374182504' 2024-09-16T07:31:06.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:06.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T07:31:06.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=365072220188 2024-09-16T07:31:06.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 365072220188 2024-09-16T07:31:06.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836605 1-42949673079 2-64424509554 3-335007449121 4-107374182504 5-365072220188' 2024-09-16T07:31:06.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:06.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-16T07:31:06.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=425201762325 2024-09-16T07:31:06.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 425201762325 2024-09-16T07:31:06.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836605 1-42949673079 2-64424509554 3-335007449121 4-107374182504 5-365072220188 6-425201762325' 2024-09-16T07:31:06.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:06.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-16T07:31:06.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=171798691921 2024-09-16T07:31:06.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 171798691921 2024-09-16T07:31:06.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836605 1-42949673079 2-64424509554 3-335007449121 4-107374182504 5-365072220188 6-425201762325 7-171798691921' 2024-09-16T07:31:06.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:06.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-16T07:31:06.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=193273528392 2024-09-16T07:31:06.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 193273528392 2024-09-16T07:31:06.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836605 1-42949673079 2-64424509554 3-335007449121 4-107374182504 5-365072220188 6-425201762325 7-171798691921 8-193273528392' 2024-09-16T07:31:06.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:06.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-16T07:31:06.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=459561500688 2024-09-16T07:31:06.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 459561500688 2024-09-16T07:31:06.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836605 1-42949673079 2-64424509554 3-335007449121 4-107374182504 5-365072220188 6-425201762325 7-171798691921 8-193273528392 9-459561500688' 2024-09-16T07:31:06.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:06.965 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-16T07:31:07.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=236223201340 2024-09-16T07:31:07.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 236223201340 2024-09-16T07:31:07.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836605 1-42949673079 2-64424509554 3-335007449121 4-107374182504 5-365072220188 6-425201762325 7-171798691921 8-193273528392 9-459561500688 10-236223201340' 2024-09-16T07:31:07.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:07.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836605 2024-09-16T07:31:07.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:07.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T07:31:07.126 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:07.126 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836605 2024-09-16T07:31:07.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836605 2024-09-16T07:31:07.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836605' 2024-09-16T07:31:07.128 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836605 2024-09-16T07:31:07.128 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:31:07.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836605 -lt 21474836605 2024-09-16T07:31:07.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:07.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673079 2024-09-16T07:31:07.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:07.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T07:31:07.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673079 2024-09-16T07:31:07.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:07.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673079 2024-09-16T07:31:07.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673079' 2024-09-16T07:31:07.598 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949673079 2024-09-16T07:31:07.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T07:31:08.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673080 -lt 42949673079 2024-09-16T07:31:08.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:08.226 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509554 2024-09-16T07:31:08.226 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:08.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T07:31:08.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509554 2024-09-16T07:31:08.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:08.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509554 2024-09-16T07:31:08.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509554' 2024-09-16T07:31:08.232 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509554 2024-09-16T07:31:08.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T07:31:08.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509555 -lt 64424509554 2024-09-16T07:31:08.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:08.715 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-335007449121 2024-09-16T07:31:08.715 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:08.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T07:31:08.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-335007449121 2024-09-16T07:31:08.719 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:08.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=335007449121 2024-09-16T07:31:08.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 335007449121' 2024-09-16T07:31:08.721 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 335007449121 2024-09-16T07:31:08.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T07:31:09.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 335007449122 -lt 335007449121 2024-09-16T07:31:09.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:09.162 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182504 2024-09-16T07:31:09.162 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:09.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T07:31:09.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182504 2024-09-16T07:31:09.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:09.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182504 2024-09-16T07:31:09.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182504' 2024-09-16T07:31:09.168 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 107374182504 2024-09-16T07:31:09.168 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T07:31:09.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182505 -lt 107374182504 2024-09-16T07:31:09.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:09.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-365072220188 2024-09-16T07:31:09.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:09.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T07:31:09.624 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-365072220188 2024-09-16T07:31:09.624 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:09.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=365072220188 2024-09-16T07:31:09.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 365072220188' 2024-09-16T07:31:09.625 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 365072220188 2024-09-16T07:31:09.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T07:31:10.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 365072220189 -lt 365072220188 2024-09-16T07:31:10.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:10.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 6-425201762325 2024-09-16T07:31:10.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:10.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=6 2024-09-16T07:31:10.085 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:10.085 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 6-425201762325 2024-09-16T07:31:10.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=425201762325 2024-09-16T07:31:10.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.6 seq 425201762325' 2024-09-16T07:31:10.086 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 425201762325 2024-09-16T07:31:10.087 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-16T07:31:10.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 425201762326 -lt 425201762325 2024-09-16T07:31:10.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:10.529 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:10.529 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 7-171798691921 2024-09-16T07:31:10.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=7 2024-09-16T07:31:10.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:10.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 7-171798691921 2024-09-16T07:31:10.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=171798691921 2024-09-16T07:31:10.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.7 seq 171798691921' 2024-09-16T07:31:10.533 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.7 seq 171798691921 2024-09-16T07:31:10.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-16T07:31:10.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 171798691921 -lt 171798691921 2024-09-16T07:31:10.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:10.978 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 8-193273528392 2024-09-16T07:31:10.978 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:10.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=8 2024-09-16T07:31:10.981 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 8-193273528392 2024-09-16T07:31:10.981 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:10.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=193273528392 2024-09-16T07:31:10.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.8 seq 193273528392' 2024-09-16T07:31:10.983 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.8 seq 193273528392 2024-09-16T07:31:10.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-16T07:31:11.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 193273528392 -lt 193273528392 2024-09-16T07:31:11.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:11.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 9-459561500688 2024-09-16T07:31:11.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:11.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=9 2024-09-16T07:31:11.391 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 9-459561500688 2024-09-16T07:31:11.392 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:11.393 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.9 seq 459561500688 2024-09-16T07:31:11.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=459561500688 2024-09-16T07:31:11.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.9 seq 459561500688' 2024-09-16T07:31:11.394 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-16T07:31:11.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 459561500689 -lt 459561500688 2024-09-16T07:31:11.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:11.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 10-236223201340 2024-09-16T07:31:11.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:11.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=10 2024-09-16T07:31:11.857 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 10-236223201340 2024-09-16T07:31:11.858 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:11.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=236223201340 2024-09-16T07:31:11.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.10 seq 236223201340' 2024-09-16T07:31:11.859 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.10 seq 236223201340 2024-09-16T07:31:11.860 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-16T07:31:12.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 236223201341 -lt 236223201340 2024-09-16T07:31:12.357 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T07:31:12.357 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:31:12.358 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:31:12.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-09-16T07:31:12.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T07:31:12.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T07:31:12.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T07:31:12.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T07:31:12.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T07:31:12.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T07:31:12.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T07:31:13.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-09-16T07:31:13.393 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T07:31:13.393 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:31:13.394 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:31:13.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-09-16T07:31:13.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T07:31:13.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T07:31:13.912 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:128: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-09-16T07:31:13.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T07:31:13.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-16T07:31:13.913 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-16T07:31:13.913 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:31:14.414 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-09-16T07:31:14.415 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:31:14.415 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-16T07:31:14.415 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:31:14.415 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:31:14.415 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-16T07:31:14.415 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 10 2024-09-16T07:31:14.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:128: rados_osds_out_in: local 'osds_list=5 1 9 4 3 10' 2024-09-16T07:31:14.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: osds=($osds_list) 2024-09-16T07:31:14.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: local -a osds 2024-09-16T07:31:14.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:130: rados_osds_out_in: for osd in 0 1 2024-09-16T07:31:14.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:131: rados_osds_out_in: ceph osd out 5 2024-09-16T07:31:14.979 INFO:tasks.workunit.client.0.smithi162.stderr:osd.5 is already out. 2024-09-16T07:31:14.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:130: rados_osds_out_in: for osd in 0 1 2024-09-16T07:31:14.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:131: rados_osds_out_in: ceph osd out 1 2024-09-16T07:31:17.538 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 is already out. 2024-09-16T07:31:17.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:133: rados_osds_out_in: wait_for_clean 2024-09-16T07:31:17.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T07:31:17.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T07:31:17.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T07:31:17.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T07:31:17.548 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T07:31:17.549 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T07:31:17.549 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T07:31:17.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T07:31:17.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T07:31:17.550 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T07:31:17.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T07:31:17.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T07:31:17.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T07:31:17.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T07:31:17.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T07:31:18.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T07:31:18.757 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:31:18.757 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:31:18.758 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:31:18.758 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:31:18.758 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:31:18.758 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-16T07:31:18.758 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-16T07:31:18.758 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-09-16T07:31:18.758 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-16T07:31:18.758 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-16T07:31:18.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T07:31:18.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:18.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T07:31:19.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836610 2024-09-16T07:31:19.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836610 2024-09-16T07:31:19.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836610' 2024-09-16T07:31:19.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:19.091 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T07:31:19.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673084 2024-09-16T07:31:19.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673084 2024-09-16T07:31:19.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836610 1-42949673084' 2024-09-16T07:31:19.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:19.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T07:31:19.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509559 2024-09-16T07:31:19.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509559 2024-09-16T07:31:19.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836610 1-42949673084 2-64424509559' 2024-09-16T07:31:19.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:19.623 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T07:31:19.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=335007449126 2024-09-16T07:31:19.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 335007449126 2024-09-16T07:31:19.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836610 1-42949673084 2-64424509559 3-335007449126' 2024-09-16T07:31:19.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:19.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T07:31:20.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182509 2024-09-16T07:31:20.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182509 2024-09-16T07:31:20.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836610 1-42949673084 2-64424509559 3-335007449126 4-107374182509' 2024-09-16T07:31:20.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:20.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T07:31:20.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=365072220193 2024-09-16T07:31:20.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 365072220193 2024-09-16T07:31:20.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836610 1-42949673084 2-64424509559 3-335007449126 4-107374182509 5-365072220193' 2024-09-16T07:31:20.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:20.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-16T07:31:20.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=425201762330 2024-09-16T07:31:20.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 425201762330 2024-09-16T07:31:20.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836610 1-42949673084 2-64424509559 3-335007449126 4-107374182509 5-365072220193 6-425201762330' 2024-09-16T07:31:20.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:20.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-16T07:31:20.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=171798691926 2024-09-16T07:31:20.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 171798691926 2024-09-16T07:31:20.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836610 1-42949673084 2-64424509559 3-335007449126 4-107374182509 5-365072220193 6-425201762330 7-171798691926' 2024-09-16T07:31:20.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:20.805 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-16T07:31:20.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=193273528397 2024-09-16T07:31:20.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 193273528397 2024-09-16T07:31:20.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836610 1-42949673084 2-64424509559 3-335007449126 4-107374182509 5-365072220193 6-425201762330 7-171798691926 8-193273528397' 2024-09-16T07:31:20.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:20.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-16T07:31:21.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=459561500693 2024-09-16T07:31:21.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 459561500693 2024-09-16T07:31:21.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836610 1-42949673084 2-64424509559 3-335007449126 4-107374182509 5-365072220193 6-425201762330 7-171798691926 8-193273528397 9-459561500693' 2024-09-16T07:31:21.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:21.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-16T07:31:21.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=236223201345 2024-09-16T07:31:21.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 236223201345 2024-09-16T07:31:21.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836610 1-42949673084 2-64424509559 3-335007449126 4-107374182509 5-365072220193 6-425201762330 7-171798691926 8-193273528397 9-459561500693 10-236223201345' 2024-09-16T07:31:21.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:21.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836610 2024-09-16T07:31:21.261 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:21.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T07:31:21.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836610 2024-09-16T07:31:21.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:21.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836610 2024-09-16T07:31:21.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836610' 2024-09-16T07:31:21.265 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836610 2024-09-16T07:31:21.266 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:31:21.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836610 -lt 21474836610 2024-09-16T07:31:21.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:21.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:21.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673084 2024-09-16T07:31:21.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T07:31:21.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673084 2024-09-16T07:31:21.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:21.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673084 2024-09-16T07:31:21.759 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949673084 2024-09-16T07:31:21.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673084' 2024-09-16T07:31:21.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T07:31:22.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673085 -lt 42949673084 2024-09-16T07:31:22.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:22.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509559 2024-09-16T07:31:22.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:22.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T07:31:22.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509559 2024-09-16T07:31:22.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:22.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509559 2024-09-16T07:31:22.203 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509559 2024-09-16T07:31:22.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509559' 2024-09-16T07:31:22.204 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T07:31:22.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509560 -lt 64424509559 2024-09-16T07:31:22.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:22.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-335007449126 2024-09-16T07:31:22.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:22.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T07:31:22.640 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-335007449126 2024-09-16T07:31:22.640 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:22.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=335007449126 2024-09-16T07:31:22.642 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 335007449126 2024-09-16T07:31:22.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 335007449126' 2024-09-16T07:31:22.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T07:31:23.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 335007449127 -lt 335007449126 2024-09-16T07:31:23.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:23.135 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182509 2024-09-16T07:31:23.135 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:23.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T07:31:23.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182509 2024-09-16T07:31:23.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:23.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182509 2024-09-16T07:31:23.141 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 107374182509 2024-09-16T07:31:23.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182509' 2024-09-16T07:31:23.142 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T07:31:23.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182509 -lt 107374182509 2024-09-16T07:31:23.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:23.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:23.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-365072220193 2024-09-16T07:31:23.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T07:31:23.569 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-365072220193 2024-09-16T07:31:23.569 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:23.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=365072220193 2024-09-16T07:31:23.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 365072220193' 2024-09-16T07:31:23.571 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 365072220193 2024-09-16T07:31:23.572 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T07:31:24.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 365072220194 -lt 365072220193 2024-09-16T07:31:24.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:24.051 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 6-425201762330 2024-09-16T07:31:24.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:24.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=6 2024-09-16T07:31:24.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:24.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 6-425201762330 2024-09-16T07:31:24.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=425201762330 2024-09-16T07:31:24.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.6 seq 425201762330' 2024-09-16T07:31:24.056 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 425201762330 2024-09-16T07:31:24.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-16T07:31:24.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 425201762331 -lt 425201762330 2024-09-16T07:31:24.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:24.529 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 7-171798691926 2024-09-16T07:31:24.529 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:24.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=7 2024-09-16T07:31:24.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 7-171798691926 2024-09-16T07:31:24.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:24.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=171798691926 2024-09-16T07:31:24.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.7 seq 171798691926' 2024-09-16T07:31:24.535 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.7 seq 171798691926 2024-09-16T07:31:24.536 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-16T07:31:24.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 171798691926 -lt 171798691926 2024-09-16T07:31:24.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:24.965 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 8-193273528397 2024-09-16T07:31:24.965 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:24.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=8 2024-09-16T07:31:24.968 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 8-193273528397 2024-09-16T07:31:24.968 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:24.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=193273528397 2024-09-16T07:31:24.969 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.8 seq 193273528397 2024-09-16T07:31:24.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.8 seq 193273528397' 2024-09-16T07:31:24.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-16T07:31:25.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 193273528397 -lt 193273528397 2024-09-16T07:31:25.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:25.425 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 9-459561500693 2024-09-16T07:31:25.425 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:25.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=9 2024-09-16T07:31:25.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:25.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 9-459561500693 2024-09-16T07:31:25.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=459561500693 2024-09-16T07:31:25.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.9 seq 459561500693' 2024-09-16T07:31:25.429 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.9 seq 459561500693 2024-09-16T07:31:25.430 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-16T07:31:25.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 459561500694 -lt 459561500693 2024-09-16T07:31:25.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:25.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:25.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 10-236223201345 2024-09-16T07:31:25.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=10 2024-09-16T07:31:25.901 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:25.902 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 10-236223201345 2024-09-16T07:31:25.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=236223201345 2024-09-16T07:31:25.903 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.10 seq 236223201345 2024-09-16T07:31:25.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.10 seq 236223201345' 2024-09-16T07:31:25.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-16T07:31:26.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 236223201346 -lt 236223201345 2024-09-16T07:31:26.314 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T07:31:26.314 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:31:26.315 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:31:26.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-09-16T07:31:26.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T07:31:26.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T07:31:26.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T07:31:26.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T07:31:26.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T07:31:26.894 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T07:31:26.894 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T07:31:27.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-09-16T07:31:27.289 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T07:31:27.289 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:31:27.289 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:31:27.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-09-16T07:31:27.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T07:31:27.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T07:31:27.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: for osd in 0 1 2024-09-16T07:31:27.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-09-16T07:31:27.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T07:31:27.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-16T07:31:27.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: grep '\<5\>' 2024-09-16T07:31:27.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-16T07:31:27.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:31:28.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-16T07:31:28.304 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-16T07:31:28.304 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-16T07:31:28.305 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:31:28.305 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:31:28.305 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-16T07:31:28.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 7 9 4 3 10 2024-09-16T07:31:28.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: for osd in 0 1 2024-09-16T07:31:28.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-09-16T07:31:28.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T07:31:28.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-16T07:31:28.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: grep '\<1\>' 2024-09-16T07:31:28.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-16T07:31:28.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:31:28.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-16T07:31:28.768 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-16T07:31:28.768 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-16T07:31:28.768 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:31:28.768 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:31:28.768 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-16T07:31:28.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 7 9 4 3 10 2024-09-16T07:31:28.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:140: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-09-16T07:31:28.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:141: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-16T07:31:28.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: for osd in 0 1 2024-09-16T07:31:28.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: ceph osd in 5 2024-09-16T07:31:29.486 INFO:tasks.workunit.client.0.smithi162.stderr:osd.5 is already in. 2024-09-16T07:31:29.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: for osd in 0 1 2024-09-16T07:31:29.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: ceph osd in 1 2024-09-16T07:31:30.880 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 is already in. 2024-09-16T07:31:30.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:150: rados_osds_out_in: wait_for_clean 2024-09-16T07:31:30.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T07:31:30.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T07:31:30.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T07:31:30.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T07:31:30.891 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T07:31:30.892 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T07:31:30.892 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T07:31:30.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T07:31:30.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T07:31:30.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T07:31:31.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T07:31:31.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T07:31:31.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T07:31:31.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T07:31:31.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T07:31:32.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T07:31:32.276 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:31:32.276 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:31:32.276 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:31:32.276 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:31:32.276 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:31:32.276 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-16T07:31:32.276 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-16T07:31:32.277 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-09-16T07:31:32.277 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-16T07:31:32.277 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-16T07:31:32.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T07:31:32.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:32.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T07:31:32.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836615 2024-09-16T07:31:32.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836615 2024-09-16T07:31:32.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836615' 2024-09-16T07:31:32.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:32.434 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T07:31:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673089 2024-09-16T07:31:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673089 2024-09-16T07:31:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836615 1-42949673089' 2024-09-16T07:31:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:32.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T07:31:33.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509564 2024-09-16T07:31:33.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509564 2024-09-16T07:31:33.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836615 1-42949673089 2-64424509564' 2024-09-16T07:31:33.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:33.169 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T07:31:33.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=335007449131 2024-09-16T07:31:33.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 335007449131 2024-09-16T07:31:33.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836615 1-42949673089 2-64424509564 3-335007449131' 2024-09-16T07:31:33.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:33.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T07:31:33.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182514 2024-09-16T07:31:33.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182514 2024-09-16T07:31:33.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836615 1-42949673089 2-64424509564 3-335007449131 4-107374182514' 2024-09-16T07:31:33.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:33.781 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T07:31:34.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=365072220198 2024-09-16T07:31:34.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 365072220198 2024-09-16T07:31:34.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836615 1-42949673089 2-64424509564 3-335007449131 4-107374182514 5-365072220198' 2024-09-16T07:31:34.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:34.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-16T07:31:34.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=425201762335 2024-09-16T07:31:34.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 425201762335 2024-09-16T07:31:34.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836615 1-42949673089 2-64424509564 3-335007449131 4-107374182514 5-365072220198 6-425201762335' 2024-09-16T07:31:34.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:34.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-16T07:31:34.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=171798691930 2024-09-16T07:31:34.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 171798691930 2024-09-16T07:31:34.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836615 1-42949673089 2-64424509564 3-335007449131 4-107374182514 5-365072220198 6-425201762335 7-171798691930' 2024-09-16T07:31:34.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:34.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-16T07:31:34.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=193273528401 2024-09-16T07:31:34.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 193273528401 2024-09-16T07:31:34.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836615 1-42949673089 2-64424509564 3-335007449131 4-107374182514 5-365072220198 6-425201762335 7-171798691930 8-193273528401' 2024-09-16T07:31:34.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:34.877 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-16T07:31:35.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=459561500697 2024-09-16T07:31:35.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 459561500697 2024-09-16T07:31:35.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836615 1-42949673089 2-64424509564 3-335007449131 4-107374182514 5-365072220198 6-425201762335 7-171798691930 8-193273528401 9-459561500697' 2024-09-16T07:31:35.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:31:35.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-16T07:31:35.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=236223201350 2024-09-16T07:31:35.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 236223201350 2024-09-16T07:31:35.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836615 1-42949673089 2-64424509564 3-335007449131 4-107374182514 5-365072220198 6-425201762335 7-171798691930 8-193273528401 9-459561500697 10-236223201350' 2024-09-16T07:31:35.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:35.226 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836615 2024-09-16T07:31:35.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:35.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T07:31:35.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836615 2024-09-16T07:31:35.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:35.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836615 2024-09-16T07:31:35.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836615' 2024-09-16T07:31:35.232 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836615 2024-09-16T07:31:35.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:31:35.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836615 -lt 21474836615 2024-09-16T07:31:35.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:35.650 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673089 2024-09-16T07:31:35.650 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:35.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T07:31:35.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:35.654 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673089 2024-09-16T07:31:35.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673089 2024-09-16T07:31:35.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673089' 2024-09-16T07:31:35.655 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949673089 2024-09-16T07:31:35.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T07:31:36.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673089 -lt 42949673089 2024-09-16T07:31:36.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:36.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509564 2024-09-16T07:31:36.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:36.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T07:31:36.155 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509564 2024-09-16T07:31:36.155 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:36.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509564 2024-09-16T07:31:36.157 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509564 2024-09-16T07:31:36.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509564' 2024-09-16T07:31:36.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T07:31:36.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509564 -lt 64424509564 2024-09-16T07:31:36.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:36.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-335007449131 2024-09-16T07:31:36.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:36.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T07:31:36.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-335007449131 2024-09-16T07:31:36.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:36.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=335007449131 2024-09-16T07:31:36.622 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 335007449131 2024-09-16T07:31:36.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 335007449131' 2024-09-16T07:31:36.623 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T07:31:37.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 335007449131 -lt 335007449131 2024-09-16T07:31:37.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:37.140 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182514 2024-09-16T07:31:37.140 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:37.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T07:31:37.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:37.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182514 2024-09-16T07:31:37.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182514 2024-09-16T07:31:37.145 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 107374182514 2024-09-16T07:31:37.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182514' 2024-09-16T07:31:37.146 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T07:31:37.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182514 -lt 107374182514 2024-09-16T07:31:37.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:37.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-365072220198 2024-09-16T07:31:37.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:37.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T07:31:37.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-365072220198 2024-09-16T07:31:37.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:37.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=365072220198 2024-09-16T07:31:37.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 365072220198' 2024-09-16T07:31:37.622 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 365072220198 2024-09-16T07:31:37.623 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T07:31:38.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 365072220198 -lt 365072220198 2024-09-16T07:31:38.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:38.062 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 6-425201762335 2024-09-16T07:31:38.062 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:38.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=6 2024-09-16T07:31:38.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 6-425201762335 2024-09-16T07:31:38.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:38.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=425201762335 2024-09-16T07:31:38.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.6 seq 425201762335' 2024-09-16T07:31:38.068 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 425201762335 2024-09-16T07:31:38.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-16T07:31:38.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 425201762336 -lt 425201762335 2024-09-16T07:31:38.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:38.537 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 7-171798691930 2024-09-16T07:31:38.537 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:38.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=7 2024-09-16T07:31:38.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:38.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 7-171798691930 2024-09-16T07:31:38.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=171798691930 2024-09-16T07:31:38.541 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.7 seq 171798691930 2024-09-16T07:31:38.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.7 seq 171798691930' 2024-09-16T07:31:38.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-16T07:31:38.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 171798691931 -lt 171798691930 2024-09-16T07:31:38.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:38.966 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 8-193273528401 2024-09-16T07:31:38.966 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:38.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=8 2024-09-16T07:31:38.969 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 8-193273528401 2024-09-16T07:31:38.969 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:38.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=193273528401 2024-09-16T07:31:38.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.8 seq 193273528401' 2024-09-16T07:31:38.972 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.8 seq 193273528401 2024-09-16T07:31:38.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-16T07:31:39.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 193273528402 -lt 193273528401 2024-09-16T07:31:39.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:39.532 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 9-459561500697 2024-09-16T07:31:39.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:39.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=9 2024-09-16T07:31:39.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 9-459561500697 2024-09-16T07:31:39.536 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:39.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=459561500697 2024-09-16T07:31:39.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.9 seq 459561500697' 2024-09-16T07:31:39.538 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.9 seq 459561500697 2024-09-16T07:31:39.538 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-16T07:31:40.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 459561500698 -lt 459561500697 2024-09-16T07:31:40.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:31:40.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 10-236223201350 2024-09-16T07:31:40.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:31:40.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=10 2024-09-16T07:31:40.099 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 10-236223201350 2024-09-16T07:31:40.099 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:31:40.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=236223201350 2024-09-16T07:31:40.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.10 seq 236223201350' 2024-09-16T07:31:40.101 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.10 seq 236223201350 2024-09-16T07:31:40.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-16T07:31:40.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 236223201351 -lt 236223201350 2024-09-16T07:31:40.558 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T07:31:40.558 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:31:40.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:31:41.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-09-16T07:31:41.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T07:31:41.142 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T07:31:41.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T07:31:41.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T07:31:41.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T07:31:41.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T07:31:41.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T07:31:41.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-09-16T07:31:41.609 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T07:31:41.609 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:31:41.610 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:31:42.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-09-16T07:31:42.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T07:31:42.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T07:31:42.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:151: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-09-16T07:31:42.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T07:31:42.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-16T07:31:42.191 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:31:42.192 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-16T07:31:42.669 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-09-16T07:31:42.669 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:31:42.669 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-16T07:31:42.669 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:31:42.669 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:31:42.669 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-16T07:31:42.669 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 10 2024-09-16T07:31:42.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:151: rados_osds_out_in: test '5 1 9 4 3 10' = '5 1 9 4 3 10' 2024-09-16T07:31:42.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:152: rados_osds_out_in: rm td/test-erasure-code/ORIGINAL 2024-09-16T07:31:42.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:229: TEST_rados_put_get_jerasure: delete_pool pool-jerasure 2024-09-16T07:31:42.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-jerasure 2024-09-16T07:31:42.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-16T07:31:43.236 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-16T07:31:43.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:230: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile rm profile-jerasure 2024-09-16T07:31:44.080 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile profile-jerasure does not exist 2024-09-16T07:31:44.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-16T07:31:44.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_advanced td/test-erasure-code 2024-09-16T07:31:44.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:156: TEST_rados_put_get_lrc_advanced: local dir=td/test-erasure-code 2024-09-16T07:31:44.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:157: TEST_rados_put_get_lrc_advanced: local poolname=pool-lrc-a 2024-09-16T07:31:44.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:158: TEST_rados_put_get_lrc_advanced: local profile=profile-lrc-a 2024-09-16T07:31:44.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:160: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile set profile-lrc-a plugin=lrc mapping=DD_ 'crush-steps=[ [ "chooseleaf", "osd", 0 ] ]' 'layers=[ [ "DDc", "" ] ]' 2024-09-16T07:31:45.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:165: TEST_rados_put_get_lrc_advanced: create_pool pool-lrc-a 12 12 erasure profile-lrc-a 2024-09-16T07:31:45.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-lrc-a 12 12 erasure profile-lrc-a 2024-09-16T07:31:46.977 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-lrc-a' already exists 2024-09-16T07:31:46.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T07:31:47.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:168: TEST_rados_put_get_lrc_advanced: rados_put_get td/test-erasure-code pool-lrc-a 2024-09-16T07:31:47.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-16T07:31:47.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc-a 2024-09-16T07:31:47.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-16T07:31:47.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-16T07:31:47.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-16T07:31:47.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-16T07:31:47.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-16T07:31:47.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-16T07:31:47.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-16T07:31:47.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-16T07:31:47.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-16T07:31:47.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc-a put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-16T07:31:48.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2024-09-16T07:31:48.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-16T07:31:48.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-16T07:31:48.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T07:31:48.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc-a SOMETHING 2024-09-16T07:31:48.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc-a 2024-09-16T07:31:48.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-16T07:31:48.681 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2024-09-16T07:31:48.681 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:31:49.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-09-16T07:31:49.762 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-16T07:31:49.762 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T07:31:49.763 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 9 3 2024-09-16T07:31:49.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-16T07:31:49.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2024-09-16T07:31:49.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 3 2024-09-16T07:31:50.620 INFO:tasks.workunit.client.0.smithi162.stderr:osd.3 is already out. 2024-09-16T07:31:50.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-09-16T07:31:55.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-lrc-a SOMETHING 2024-09-16T07:31:55.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc-a 2024-09-16T07:31:55.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-16T07:31:55.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<3\>' 2024-09-16T07:31:55.636 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2024-09-16T07:31:55.636 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:31:56.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-09-16T07:31:56.125 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-16T07:31:56.125 INFO:tasks.workunit.client.0.smithi162.stderr:7' 2024-09-16T07:31:56.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 9 7 2024-09-16T07:31:56.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2024-09-16T07:31:56.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-16T07:31:56.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 3 2024-09-16T07:31:57.036 INFO:tasks.workunit.client.0.smithi162.stderr:osd.3 is already in. 2024-09-16T07:31:57.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-16T07:31:57.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:170: TEST_rados_put_get_lrc_advanced: delete_pool pool-lrc-a 2024-09-16T07:31:57.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-lrc-a 2024-09-16T07:31:57.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-lrc-a pool-lrc-a --yes-i-really-really-mean-it 2024-09-16T07:31:58.455 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-lrc-a' does not exist 2024-09-16T07:31:58.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:171: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile rm profile-lrc-a 2024-09-16T07:31:59.612 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile profile-lrc-a does not exist 2024-09-16T07:31:59.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-16T07:31:59.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_kml td/test-erasure-code 2024-09-16T07:31:59.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:175: TEST_rados_put_get_lrc_kml: local dir=td/test-erasure-code 2024-09-16T07:31:59.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:176: TEST_rados_put_get_lrc_kml: local poolname=pool-lrc 2024-09-16T07:31:59.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:177: TEST_rados_put_get_lrc_kml: local profile=profile-lrc 2024-09-16T07:31:59.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:179: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile set profile-lrc plugin=lrc k=4 m=2 l=3 crush-failure-domain=osd 2024-09-16T07:32:01.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:183: TEST_rados_put_get_lrc_kml: create_pool pool-lrc 12 12 erasure profile-lrc 2024-09-16T07:32:01.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-lrc 12 12 erasure profile-lrc 2024-09-16T07:32:02.092 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-lrc' already exists 2024-09-16T07:32:02.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T07:32:03.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:186: TEST_rados_put_get_lrc_kml: rados_put_get td/test-erasure-code pool-lrc 2024-09-16T07:32:03.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-16T07:32:03.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc 2024-09-16T07:32:03.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-16T07:32:03.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-16T07:32:03.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-16T07:32:03.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-16T07:32:03.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-16T07:32:03.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-16T07:32:03.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-16T07:32:03.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-16T07:32:03.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-16T07:32:03.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-16T07:32:04.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2024-09-16T07:32:04.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-16T07:32:04.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-16T07:32:04.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T07:32:04.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc SOMETHING 2024-09-16T07:32:04.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc 2024-09-16T07:32:04.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-16T07:32:04.719 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc SOMETHING 2024-09-16T07:32:04.719 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:32:05.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=8 2024-09-16T07:32:05.365 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-16T07:32:05.365 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T07:32:05.365 INFO:tasks.workunit.client.0.smithi162.stderr:10 2024-09-16T07:32:05.365 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:32:05.365 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:32:05.365 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-16T07:32:05.365 INFO:tasks.workunit.client.0.smithi162.stderr:7' 2024-09-16T07:32:05.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 8 6 0 10 5 3 9 7 2024-09-16T07:32:05.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-16T07:32:05.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=7 2024-09-16T07:32:05.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 7 2024-09-16T07:32:06.010 INFO:tasks.workunit.client.0.smithi162.stderr:osd.7 is already out. 2024-09-16T07:32:06.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-09-16T07:32:11.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-lrc SOMETHING 2024-09-16T07:32:11.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc 2024-09-16T07:32:11.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<7\>' 2024-09-16T07:32:11.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-16T07:32:11.024 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc SOMETHING 2024-09-16T07:32:11.024 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:32:11.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=8 2024-09-16T07:32:11.453 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-16T07:32:11.453 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T07:32:11.453 INFO:tasks.workunit.client.0.smithi162.stderr:10 2024-09-16T07:32:11.453 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:32:11.453 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:32:11.453 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-16T07:32:11.453 INFO:tasks.workunit.client.0.smithi162.stderr:4' 2024-09-16T07:32:11.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 8 6 0 10 5 3 9 4 2024-09-16T07:32:11.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2024-09-16T07:32:11.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-16T07:32:11.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 7 2024-09-16T07:32:12.566 INFO:tasks.workunit.client.0.smithi162.stderr:osd.7 is already in. 2024-09-16T07:32:12.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-16T07:32:12.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:188: TEST_rados_put_get_lrc_kml: delete_pool pool-lrc 2024-09-16T07:32:12.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-lrc 2024-09-16T07:32:12.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-lrc pool-lrc --yes-i-really-really-mean-it 2024-09-16T07:32:14.445 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-lrc' does not exist 2024-09-16T07:32:14.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:189: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile rm profile-lrc 2024-09-16T07:32:16.014 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile profile-lrc does not exist 2024-09-16T07:32:16.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-16T07:32:16.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_shec td/test-erasure-code 2024-09-16T07:32:16.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:234: TEST_rados_put_get_shec: local dir=td/test-erasure-code 2024-09-16T07:32:16.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:236: TEST_rados_put_get_shec: local poolname=pool-shec 2024-09-16T07:32:16.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:237: TEST_rados_put_get_shec: local profile=profile-shec 2024-09-16T07:32:16.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:239: TEST_rados_put_get_shec: ceph osd erasure-code-profile set profile-shec plugin=shec k=2 m=1 c=1 crush-failure-domain=osd 2024-09-16T07:32:16.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:243: TEST_rados_put_get_shec: create_pool pool-shec 12 12 erasure profile-shec 2024-09-16T07:32:16.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-shec 12 12 erasure profile-shec 2024-09-16T07:32:17.801 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-shec' already exists 2024-09-16T07:32:17.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T07:32:18.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:246: TEST_rados_put_get_shec: rados_put_get td/test-erasure-code pool-shec 2024-09-16T07:32:18.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-16T07:32:18.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-shec 2024-09-16T07:32:18.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-16T07:32:18.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-16T07:32:18.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-16T07:32:18.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-16T07:32:18.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-16T07:32:18.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-16T07:32:18.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-16T07:32:18.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-16T07:32:18.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-16T07:32:18.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-shec put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-16T07:32:20.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2024-09-16T07:32:20.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-16T07:32:20.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-16T07:32:20.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T07:32:20.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-shec SOMETHING 2024-09-16T07:32:20.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-shec 2024-09-16T07:32:20.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-16T07:32:20.614 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-shec SOMETHING 2024-09-16T07:32:20.614 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:32:21.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=7 2024-09-16T07:32:21.148 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-16T07:32:21.149 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-09-16T07:32:21.149 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 7 9 1 2024-09-16T07:32:21.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-16T07:32:21.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2024-09-16T07:32:21.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 1 2024-09-16T07:32:21.777 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 is already out. 2024-09-16T07:32:21.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-09-16T07:32:26.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-shec SOMETHING 2024-09-16T07:32:26.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-shec 2024-09-16T07:32:26.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-16T07:32:26.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<1\>' 2024-09-16T07:32:26.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-shec SOMETHING 2024-09-16T07:32:26.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:32:27.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=7 2024-09-16T07:32:27.230 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-16T07:32:27.230 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-16T07:32:27.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 7 9 6 2024-09-16T07:32:27.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2024-09-16T07:32:27.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-16T07:32:27.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 1 2024-09-16T07:32:27.891 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 is already in. 2024-09-16T07:32:27.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-16T07:32:27.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:248: TEST_rados_put_get_shec: delete_pool pool-shec 2024-09-16T07:32:27.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-shec 2024-09-16T07:32:27.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-shec pool-shec --yes-i-really-really-mean-it 2024-09-16T07:32:29.578 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-shec' does not exist 2024-09-16T07:32:29.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:249: TEST_rados_put_get_shec: ceph osd erasure-code-profile rm profile-shec 2024-09-16T07:32:31.286 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile profile-shec does not exist 2024-09-16T07:32:31.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:51: run: delete_pool ecpool 2024-09-16T07:32:31.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=ecpool 2024-09-16T07:32:31.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete ecpool ecpool --yes-i-really-really-mean-it 2024-09-16T07:32:32.195 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'ecpool' does not exist 2024-09-16T07:32:32.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:52: run: teardown td/test-erasure-code 2024-09-16T07:32:32.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2024-09-16T07:32:32.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:32:32.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2024-09-16T07:32:32.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:32:32.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:32:32.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:32:32.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:32:32.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:32:32.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:32:32.536 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:32:32.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:32:32.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:32:32.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:32:32.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:32:32.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:32:32.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:32:32.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:32:32.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:32:32.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:32:32.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:32:32.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:32:32.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2024-09-16T07:32:32.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:32:32.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:32:32.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:32:32.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.46203 2024-09-16T07:32:32.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:32:32.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:32:32.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T07:32:32.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/test-erasure-code 0 2024-09-16T07:32:32.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2024-09-16T07:32:32.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T07:32:32.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2024-09-16T07:32:32.646 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:32:32.646 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:32:32.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:32:32.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:32:32.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:32:32.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:32:32.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:32:32.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:32:32.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:32:32.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:32:32.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:32:32.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:32:32.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:32:32.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:32:32.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:32:32.658 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:32:32.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:32:32.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T07:32:32.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2024-09-16T07:32:32.664 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:32:32.664 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:32:32.664 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46203 2024-09-16T07:32:32.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.46203 2024-09-16T07:32:32.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:32:32.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:32:32.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T07:32:32.667 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T07:32:32.667 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T07:32:32.683 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-eio.sh... 2024-09-16T07:32:32.683 DEBUG:teuthology.orchestra.run.smithi162:workunit test erasure-code/test-erasure-eio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh 2024-09-16T07:32:32.743 INFO:tasks.workunit.client.0.smithi162.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T07:32:32.757 INFO:tasks.workunit.client.0.smithi162.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T07:32:32.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T07:32:32.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T07:32:32.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T07:32:32.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T07:32:32.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T07:32:32.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T07:32:32.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T07:32:32.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T07:32:32.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/test-erasure-eio 2024-09-16T07:32:32.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:22: run: local dir=td/test-erasure-eio 2024-09-16T07:32:32.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:23: run: shift 2024-09-16T07:32:32.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: export CEPH_MON=127.0.0.1:7112 2024-09-16T07:32:32.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: CEPH_MON=127.0.0.1:7112 2024-09-16T07:32:32.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:26: run: export CEPH_ARGS 2024-09-16T07:32:32.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: uuidgen 2024-09-16T07:32:32.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: CEPH_ARGS+='--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none ' 2024-09-16T07:32:32.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7112 ' 2024-09-16T07:32:32.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:29: run: CEPH_ARGS+='--osd_mclock_override_recovery_settings=true ' 2024-09-16T07:32:32.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: set 2024-09-16T07:32:32.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T07:32:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: local 'funcs=TEST_ec_backfill_unfound 2024-09-16T07:32:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_ec_object_attr_read_error 2024-09-16T07:32:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_ec_recovery_multiple_errors 2024-09-16T07:32:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_ec_recovery_multiple_objects 2024-09-16T07:32:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_ec_recovery_multiple_objects_eio 2024-09-16T07:32:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_ec_recovery_unfound 2024-09-16T07:32:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_ec_single_recovery_error 2024-09-16T07:32:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_rados_get_bad_size_shard_0 2024-09-16T07:32:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_rados_get_bad_size_shard_1 2024-09-16T07:32:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_rados_get_subread_eio_shard_0 2024-09-16T07:32:32.765 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_rados_get_subread_eio_shard_1 2024-09-16T07:32:32.765 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_rados_get_subread_missing 2024-09-16T07:32:32.765 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_rados_get_with_subreadall_eio_shard_0 2024-09-16T07:32:32.765 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_rados_get_with_subreadall_eio_shard_1' 2024-09-16T07:32:32.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-16T07:32:32.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-16T07:32:32.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-16T07:32:32.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-16T07:32:32.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T07:32:32.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:32:32.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T07:32:32.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:32:32.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:32:32.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:32:32.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:32:32.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:32:32.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:32:32.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:32:32.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:32:32.770 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:32:32.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:32:32.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:32:32.772 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:32:32.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:32:32.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:32:32.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:32:32.775 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:32:32.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:32:32.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:32:32.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T07:32:32.780 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:32:32.780 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:32:32.780 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:32:32.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T07:32:32.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:32:32.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:32:32.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-16T07:32:32.785 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T07:32:32.785 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:32:32.785 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:32:32.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79228 2024-09-16T07:32:32.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T07:32:32.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T07:32:32.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T07:32:32.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T07:32:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-16T07:32:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-16T07:32:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-16T07:32:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T07:32:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T07:32:32.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T07:32:32.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-16T07:32:32.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-16T07:32:33.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T07:32:33.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:32:33.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:32:33.211 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:32:33.212 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:32:33.212 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:32:33.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:32:33.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T07:32:33.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T07:32:33.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T07:32:33.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:32:33.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:32:33.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T07:32:33.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T07:32:33.474 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:32:33.474 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:32:33.474 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:32:33.475 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:32:33.475 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:32:33.475 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:32:33.475 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T07:32:33.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:32:33.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get fsid 2024-09-16T07:32:33.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T07:32:33.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:32:33.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:32:33.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T07:32:33.588 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T07:32:33.588 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:32:33.588 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:32:33.588 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:32:33.588 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:32:33.588 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:32:33.589 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:32:33.589 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T07:32:33.589 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:32:33.590 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get mon_host 2024-09-16T07:32:33.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-16T07:32:33.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-16T07:32:33.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T07:32:33.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T07:32:33.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T07:32:33.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-16T07:32:33.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T07:32:34.593 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T07:32:34.593 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:32:34.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:32:34.594 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:32:34.594 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:32:34.594 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:32:34.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:32:34.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:32:34.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:32:34.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-16T07:32:34.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T07:32:34.947 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-16T07:32:34.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T07:32:35.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-16T07:32:35.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:32:35.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:32:35.960 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:32:35.960 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:32:35.960 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:32:35.960 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T07:32:35.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-16T07:32:35.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-mon.a.asok log flush 2024-09-16T07:32:36.045 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:32:36.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-16T07:32:36.056 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:32:33.265+0000 7f29d7733d40 0 load: jerasure load: lrc 2024-09-16T07:32:36.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_backfill_unfound td/test-erasure-eio 2024-09-16T07:32:36.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:511: TEST_ec_backfill_unfound: local dir=td/test-erasure-eio 2024-09-16T07:32:36.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:512: TEST_ec_backfill_unfound: local objname=myobject 2024-09-16T07:32:36.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:513: TEST_ec_backfill_unfound: local lastobj=300 2024-09-16T07:32:36.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:515: TEST_ec_backfill_unfound: local testobj=obj250 2024-09-16T07:32:36.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:517: TEST_ec_backfill_unfound: ORIG_ARGS='--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:32:36.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:518: TEST_ec_backfill_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-16T07:32:36.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:519: TEST_ec_backfill_unfound: setup_osds 5 2024-09-16T07:32:36.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2024-09-16T07:32:36.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-16T07:32:36.058 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2024-09-16T07:32:36.058 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2024-09-16T07:32:36.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:32:36.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-16T07:32:36.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:32:36.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:32:36.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T07:32:36.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:32:36.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-16T07:32:36.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-16T07:32:36.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:32:36.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:32:36.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:32:36.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-16T07:32:36.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-16T07:32:36.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:32:36.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:32:36.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:32:36.062 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:32:36.062 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:32:36.062 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:32:36.064 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:32:36.064 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:32:36.064 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:32:36.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:32:36.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:32:36.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:32:36.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:32:36.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:32:36.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:32:36.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:32:36.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:32:36.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:32:36.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:32:36.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:32:36.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:32:36.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:32:36.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-16T07:32:36.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:32:36.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5b1fb133-25b5-47ad-9095-29c5bdcdcce8 2024-09-16T07:32:36.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5b1fb133-25b5-47ad-9095-29c5bdcdcce8' 2024-09-16T07:32:36.067 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 5b1fb133-25b5-47ad-9095-29c5bdcdcce8 2024-09-16T07:32:36.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:32:36.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAU3+dm39eQBRAArpM+ghS/G8MAfmE3dSu+qg== 2024-09-16T07:32:36.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAU3+dm39eQBRAArpM+ghS/G8MAfmE3dSu+qg=="}' 2024-09-16T07:32:36.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5b1fb133-25b5-47ad-9095-29c5bdcdcce8 -i td/test-erasure-eio/0/new.json 2024-09-16T07:32:36.422 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:32:36.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-16T07:32:36.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAU3+dm39eQBRAArpM+ghS/G8MAfmE3dSu+qg== --osd-uuid 5b1fb133-25b5-47ad-9095-29c5bdcdcce8 2024-09-16T07:32:36.475 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:32:36.473+0000 7f22c511d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:32:36.484 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:32:36.481+0000 7f22c511d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:32:36.485 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:32:36.485+0000 7f22c511d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:32:36.486 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:32:36.485+0000 7f22c511d880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T07:32:36.486 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:32:36.485+0000 7f22c511d880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T07:32:36.487 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:32:36.485+0000 7f22c511d880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T07:32:36.524 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:32:36.521+0000 7f22c511d880 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-16T07:32:42.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-16T07:32:42.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:32:42.317 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T07:32:42.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T07:32:42.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:32:42.881 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T07:32:42.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T07:32:42.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:32:42.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:32:42.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:32:42.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:32:42.919 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:32:42.917+0000 7faf18640880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:32:42.948 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:32:42.945+0000 7faf18640880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:32:42.949 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:32:42.949+0000 7faf18640880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:32:43.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T07:32:43.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:32:43.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T07:32:43.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:32:43.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:32:43.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:32:43.323 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:32:43.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:32:43.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:32:43.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:32:43.516 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:32:43.513+0000 7faf18640880 -1 Falling back to public interface 2024-09-16T07:32:43.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:32:44.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:32:44.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:32:44.762 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:32:44.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:32:44.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:32:44.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:32:45.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:32:45.873 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:32:45.869+0000 7faf18640880 -1 osd.0 0 log_to_monitors true 2024-09-16T07:32:46.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:32:46.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:32:46.190 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:32:46.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:32:46.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:32:46.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:32:46.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:32:47.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:32:47.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:32:47.839 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:32:47.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:32:47.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:32:47.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:32:48.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:32:49.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:32:49.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:32:49.299 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:32:49.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:32:49.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:32:49.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:32:49.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:32:50.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:32:50.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:32:50.753 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:32:50.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:32:50.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:32:50.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:32:51.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:32:52.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:32:52.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:32:52.173 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:32:52.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:32:52.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:32:52.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:32:52.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:32:53.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:32:53.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:32:53.773 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:32:53.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:32:53.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:32:53.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:32:54.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:32:55.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:32:55.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:32:55.230 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:32:55.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:32:55.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:32:55.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:32:55.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:32:56.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:32:56.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:32:56.749 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:32:56.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:32:56.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:32:56.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:32:57.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:32:58.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:32:58.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:32:58.198 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:32:58.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:32:58.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:32:58.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:32:58.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:32:59.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:32:59.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:32:59.748 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:32:59.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:32:59.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:32:59.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:33:00.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:01.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:01.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:01.701 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:33:01.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:33:01.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:01.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:33:02.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:03.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:03.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:03.281 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:33:03.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:33:03.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:03.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:33:04.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:05.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:05.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:05.153 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:33:05.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:33:05.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:05.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:33:06.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:07.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:07.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:07.099 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:33:07.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:33:07.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:07.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:33:07.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:08.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:08.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:08.981 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:33:08.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:33:08.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:08.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:33:09.540 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2552249308,v1:127.0.0.1:6803/2552249308] [v2:127.0.0.1:6804/2552249308,v1:127.0.0.1:6805/2552249308] exists,up 5b1fb133-25b5-47ad-9095-29c5bdcdcce8 2024-09-16T07:33:09.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:33:09.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:33:09.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:33:09.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:33:09.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-16T07:33:09.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:33:09.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:33:09.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T07:33:09.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:33:09.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-16T07:33:09.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-16T07:33:09.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:33:09.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:33:09.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:33:09.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-16T07:33:09.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-16T07:33:09.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:33:09.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:33:09.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:33:09.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:33:09.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:33:09.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:33:09.544 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:33:09.544 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:33:09.544 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:33:09.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:33:09.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:33:09.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:33:09.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:33:09.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:33:09.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:33:09.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:33:09.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:33:09.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:33:09.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:33:09.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:33:09.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:33:09.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:33:09.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-16T07:33:09.547 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:33:09.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7ecf1f7a-1f88-4fe0-a798-370ac019d610 2024-09-16T07:33:09.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7ecf1f7a-1f88-4fe0-a798-370ac019d610' 2024-09-16T07:33:09.548 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 7ecf1f7a-1f88-4fe0-a798-370ac019d610 2024-09-16T07:33:09.548 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:33:09.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA13+dmPTJiIhAA65HPNybDkyXSLyJfu+iKQw== 2024-09-16T07:33:09.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA13+dmPTJiIhAA65HPNybDkyXSLyJfu+iKQw=="}' 2024-09-16T07:33:09.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7ecf1f7a-1f88-4fe0-a798-370ac019d610 -i td/test-erasure-eio/1/new.json 2024-09-16T07:33:10.257 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:33:10.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-16T07:33:10.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA13+dmPTJiIhAA65HPNybDkyXSLyJfu+iKQw== --osd-uuid 7ecf1f7a-1f88-4fe0-a798-370ac019d610 2024-09-16T07:33:10.317 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:33:10.312+0000 7f0b8fac7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:33:10.325 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:33:10.320+0000 7f0b8fac7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:33:10.327 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:33:10.324+0000 7f0b8fac7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:33:10.328 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:33:10.324+0000 7f0b8fac7880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T07:33:10.328 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:33:10.324+0000 7f0b8fac7880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T07:33:10.328 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:33:10.324+0000 7f0b8fac7880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T07:33:10.425 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:33:10.420+0000 7f0b8fac7880 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-16T07:33:16.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-16T07:33:16.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:33:16.272 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-16T07:33:16.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T07:33:16.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:33:16.822 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-16T07:33:16.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T07:33:16.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:33:16.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:33:16.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:33:16.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:33:16.862 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:33:16.860+0000 7fa687376880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:33:16.879 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:33:16.876+0000 7fa687376880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:33:16.881 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:33:16.876+0000 7fa687376880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:33:17.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T07:33:17.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:33:17.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T07:33:17.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:33:17.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:33:17.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:17.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:33:17.252 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:33:17.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:17.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:33:17.455 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:33:17.452+0000 7fa687376880 -1 Falling back to public interface 2024-09-16T07:33:17.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:18.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:18.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:18.715 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:33:18.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:33:18.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:18.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:33:19.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:19.846 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:33:19.844+0000 7fa687376880 -1 osd.1 0 log_to_monitors true 2024-09-16T07:33:20.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:20.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:20.141 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:33:20.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:33:20.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:20.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:33:20.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:21.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:21.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:21.677 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:33:21.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:33:21.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:21.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:33:22.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:23.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:23.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:23.198 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:33:23.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:33:23.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:23.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:33:23.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:24.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:24.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:24.781 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:33:24.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:33:24.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:24.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:33:25.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:26.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:26.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:26.351 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:33:26.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:33:26.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:26.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:33:26.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:27.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:27.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:27.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:33:27.826 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:33:27.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:27.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:33:28.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:29.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:29.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:29.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:33:29.267 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:33:29.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:29.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:33:29.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:30.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:30.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:30.797 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:33:30.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:33:30.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:30.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:33:31.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:32.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:32.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:32.384 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:33:32.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:33:32.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:32.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:33:33.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:34.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:34.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:34.316 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:33:34.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:33:34.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:34.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:33:35.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:36.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:36.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:36.384 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:33:36.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:33:36.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:36.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:33:37.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:38.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:38.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:38.182 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:33:38.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:33:38.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:38.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:33:39.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:40.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:40.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:40.048 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:33:40.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:33:40.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:40.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:33:40.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:41.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:41.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:41.999 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:33:42.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:33:42.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:42.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:33:43.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:44.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:44.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:44.221 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:33:44.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:33:44.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:44.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:33:44.733 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3072140902,v1:127.0.0.1:6811/3072140902] [v2:127.0.0.1:6812/3072140902,v1:127.0.0.1:6813/3072140902] exists,up 7ecf1f7a-1f88-4fe0-a798-370ac019d610 2024-09-16T07:33:44.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:33:44.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:33:44.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:33:44.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:33:44.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-16T07:33:44.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:33:44.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:33:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T07:33:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:33:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-16T07:33:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-16T07:33:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:33:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:33:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:33:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-16T07:33:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-16T07:33:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:33:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:33:44.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:33:44.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:33:44.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:33:44.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:33:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:33:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:33:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:33:44.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:33:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:33:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:33:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:33:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:33:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:33:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:33:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:33:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:33:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:33:44.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:33:44.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:33:44.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:33:44.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-16T07:33:44.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:33:44.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f6fd136e-d706-4bc6-a2ca-2b71afe2aa2d 2024-09-16T07:33:44.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f6fd136e-d706-4bc6-a2ca-2b71afe2aa2d' 2024-09-16T07:33:44.741 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 f6fd136e-d706-4bc6-a2ca-2b71afe2aa2d 2024-09-16T07:33:44.741 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:33:44.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBY3+dm/rbILRAAsj85Y78XMztBQLve3DPyyw== 2024-09-16T07:33:44.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBY3+dm/rbILRAAsj85Y78XMztBQLve3DPyyw=="}' 2024-09-16T07:33:44.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f6fd136e-d706-4bc6-a2ca-2b71afe2aa2d -i td/test-erasure-eio/2/new.json 2024-09-16T07:33:45.468 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:33:45.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-16T07:33:45.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBY3+dm/rbILRAAsj85Y78XMztBQLve3DPyyw== --osd-uuid f6fd136e-d706-4bc6-a2ca-2b71afe2aa2d 2024-09-16T07:33:45.529 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:33:45.523+0000 7fac73bbd880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:33:45.548 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:33:45.543+0000 7fac73bbd880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:33:45.550 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:33:45.547+0000 7fac73bbd880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:33:45.551 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:33:45.547+0000 7fac73bbd880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T07:33:45.551 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:33:45.547+0000 7fac73bbd880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T07:33:45.551 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:33:45.547+0000 7fac73bbd880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T07:33:45.640 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:33:45.635+0000 7fac73bbd880 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-16T07:33:52.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-16T07:33:52.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:33:52.229 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-16T07:33:52.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T07:33:52.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:33:52.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T07:33:52.810 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-16T07:33:52.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:33:52.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:33:52.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:33:52.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:33:52.848 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:33:52.843+0000 7fa02a776880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:33:52.855 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:33:52.851+0000 7fa02a776880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:33:52.857 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:33:52.851+0000 7fa02a776880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:33:53.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T07:33:53.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:33:53.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T07:33:53.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:33:53.230 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:33:53.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:33:53.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:53.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:33:53.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:53.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:33:53.431 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:33:53.427+0000 7fa02a776880 -1 Falling back to public interface 2024-09-16T07:33:53.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:54.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:54.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:54.657 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:33:54.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:33:54.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:54.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:33:55.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:55.866 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:33:55.863+0000 7fa02a776880 -1 osd.2 0 log_to_monitors true 2024-09-16T07:33:56.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:56.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:56.104 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:33:56.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:33:56.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:56.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:33:56.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:57.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:57.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:57.679 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:33:57.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:33:57.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:57.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:33:58.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:33:59.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:33:59.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:33:59.141 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:33:59.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:33:59.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:33:59.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:33:59.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:00.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:00.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:00.642 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:34:00.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:34:00.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:00.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:34:01.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:02.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:02.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:02.093 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:34:02.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:34:02.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:02.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:34:02.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:03.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:03.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:03.662 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:34:03.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:34:03.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:03.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:34:04.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:05.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:05.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:05.094 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:34:05.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:34:05.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:05.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:34:05.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:06.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:06.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:06.580 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:34:06.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:34:06.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:06.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:34:07.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:08.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:08.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:08.121 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:34:08.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:34:08.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:08.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:34:09.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:10.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:10.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:10.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:34:10.144 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:34:10.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:10.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:34:11.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:12.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:12.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:12.179 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:34:12.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:34:12.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:34:12.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:13.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:14.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:14.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:14.135 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:34:14.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:34:14.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:14.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:34:14.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:15.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:15.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:15.943 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:34:15.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:34:15.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:15.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:34:16.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:17.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:17.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:17.591 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:34:17.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:34:17.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:17.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:34:18.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:19.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:19.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:19.753 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:34:19.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:34:19.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:19.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:34:20.283 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/402160535,v1:127.0.0.1:6819/402160535] [v2:127.0.0.1:6820/402160535,v1:127.0.0.1:6821/402160535] exists,up f6fd136e-d706-4bc6-a2ca-2b71afe2aa2d 2024-09-16T07:34:20.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:34:20.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:34:20.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:34:20.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:34:20.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-16T07:34:20.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:34:20.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:34:20.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T07:34:20.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:34:20.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-16T07:34:20.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-16T07:34:20.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:34:20.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:34:20.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:34:20.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-16T07:34:20.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-16T07:34:20.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:34:20.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:34:20.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:34:20.287 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:34:20.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:34:20.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:34:20.288 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:34:20.288 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:34:20.288 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:34:20.289 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:34:20.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:34:20.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:34:20.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:34:20.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:34:20.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:34:20.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:34:20.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:34:20.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:34:20.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:34:20.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:34:20.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:34:20.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:34:20.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-16T07:34:20.291 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:34:20.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e2a2959e-03d8-44ab-91b9-f6d795a81819 2024-09-16T07:34:20.293 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 e2a2959e-03d8-44ab-91b9-f6d795a81819 2024-09-16T07:34:20.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 e2a2959e-03d8-44ab-91b9-f6d795a81819' 2024-09-16T07:34:20.294 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:34:20.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB83+dm9EMHExAAT98dfm1iwMTZ4R+eG/ufFA== 2024-09-16T07:34:20.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB83+dm9EMHExAAT98dfm1iwMTZ4R+eG/ufFA=="}' 2024-09-16T07:34:20.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e2a2959e-03d8-44ab-91b9-f6d795a81819 -i td/test-erasure-eio/3/new.json 2024-09-16T07:34:21.173 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:34:21.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-16T07:34:21.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB83+dm9EMHExAAT98dfm1iwMTZ4R+eG/ufFA== --osd-uuid e2a2959e-03d8-44ab-91b9-f6d795a81819 2024-09-16T07:34:21.233 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:34:21.227+0000 7f8adacf1880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:34:21.246 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:34:21.243+0000 7f8adacf1880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:34:21.247 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:34:21.243+0000 7f8adacf1880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:34:21.248 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:34:21.243+0000 7f8adacf1880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T07:34:21.248 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:34:21.243+0000 7f8adacf1880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T07:34:21.248 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:34:21.243+0000 7f8adacf1880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T07:34:21.343 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:34:21.343+0000 7f8adacf1880 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-16T07:34:28.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-16T07:34:28.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:34:28.084 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-16T07:34:28.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T07:34:28.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:34:28.657 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-16T07:34:28.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T07:34:28.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:34:28.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:34:28.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:34:28.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:34:28.696 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:34:28.690+0000 7fae3e669880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:34:28.714 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:34:28.710+0000 7fae3e669880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:34:28.716 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:34:28.710+0000 7fae3e669880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:34:29.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T07:34:29.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:34:29.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T07:34:29.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:34:29.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:34:29.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:29.143 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:34:29.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:34:29.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:29.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:34:29.280 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:34:29.274+0000 7fae3e669880 -1 Falling back to public interface 2024-09-16T07:34:29.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:30.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:30.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:30.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:34:30.611 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:34:30.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:34:30.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:31.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:31.545 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:34:31.538+0000 7fae3e669880 -1 osd.3 0 log_to_monitors true 2024-09-16T07:34:32.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:32.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:32.076 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:34:32.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:34:32.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:32.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:34:32.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:33.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:33.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:33.945 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:34:33.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:34:33.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:33.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:34:34.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:35.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:35.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:35.695 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:34:35.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:34:35.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:35.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:34:36.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:37.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:37.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:37.131 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:34:37.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:34:37.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:37.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:34:37.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:38.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:38.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:38.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:34:38.607 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:34:38.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:38.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:34:39.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:40.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:40.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:40.121 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:34:40.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:34:40.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:40.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:34:40.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:41.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:41.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:41.723 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:34:41.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:34:41.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:41.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:34:42.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:43.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:43.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:43.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:34:43.134 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:34:43.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:43.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:34:43.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:44.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:44.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:44.593 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:34:44.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:34:44.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:44.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:34:45.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:46.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:46.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:46.350 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:34:46.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:34:46.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:46.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:34:47.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:48.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:48.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:48.072 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:34:48.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:34:48.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:48.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:34:48.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:49.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:49.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:49.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:34:49.834 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:34:49.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:49.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:34:50.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:51.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:51.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:51.599 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:34:51.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:34:51.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:51.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:34:52.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:53.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:53.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:53.728 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:34:53.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:34:53.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:53.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:34:54.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:55.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:55.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:55.570 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:34:55.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:34:55.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:55.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:34:56.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:34:57.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:34:57.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:34:57.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:34:57.709 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:34:57.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:34:57.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:34:58.421 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4041598249,v1:127.0.0.1:6827/4041598249] [v2:127.0.0.1:6828/4041598249,v1:127.0.0.1:6829/4041598249] exists,up e2a2959e-03d8-44ab-91b9-f6d795a81819 2024-09-16T07:34:58.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:34:58.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:34:58.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:34:58.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:34:58.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-16T07:34:58.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:34:58.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:34:58.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-16T07:34:58.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:34:58.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-16T07:34:58.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-16T07:34:58.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:34:58.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:34:58.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:34:58.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-16T07:34:58.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-16T07:34:58.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:34:58.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:34:58.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:34:58.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:34:58.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:34:58.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:34:58.427 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:34:58.427 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:34:58.427 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:34:58.427 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:34:58.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:34:58.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:34:58.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:34:58.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:34:58.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:34:58.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:34:58.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:34:58.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:34:58.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:34:58.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:34:58.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:34:58.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:34:58.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-16T07:34:58.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:34:58.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d5ac2151-dfcc-4bef-ac20-82303ffee290 2024-09-16T07:34:58.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 d5ac2151-dfcc-4bef-ac20-82303ffee290' 2024-09-16T07:34:58.430 INFO:tasks.workunit.client.0.smithi162.stdout:add osd4 d5ac2151-dfcc-4bef-ac20-82303ffee290 2024-09-16T07:34:58.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:34:58.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCi3+dmmNhGGxAAZJppkjMzUm89co3lWJeCkA== 2024-09-16T07:34:58.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCi3+dmmNhGGxAAZJppkjMzUm89co3lWJeCkA=="}' 2024-09-16T07:34:58.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d5ac2151-dfcc-4bef-ac20-82303ffee290 -i td/test-erasure-eio/4/new.json 2024-09-16T07:34:59.273 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:34:59.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-16T07:34:59.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCi3+dmmNhGGxAAZJppkjMzUm89co3lWJeCkA== --osd-uuid d5ac2151-dfcc-4bef-ac20-82303ffee290 2024-09-16T07:34:59.330 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:34:59.330+0000 7f09f7f70880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:34:59.388 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:34:59.386+0000 7f09f7f70880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:34:59.390 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:34:59.390+0000 7f09f7f70880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:34:59.391 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:34:59.390+0000 7f09f7f70880 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-16T07:34:59.391 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:34:59.390+0000 7f09f7f70880 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-16T07:34:59.391 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:34:59.390+0000 7f09f7f70880 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-16T07:34:59.591 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:34:59.590+0000 7f09f7f70880 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-16T07:35:06.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-16T07:35:06.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:35:06.085 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd4 key to auth repository 2024-09-16T07:35:06.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-16T07:35:06.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:35:06.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-16T07:35:06.698 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.4 2024-09-16T07:35:06.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:35:06.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:35:06.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:35:06.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:35:06.737 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:35:06.733+0000 7f7b1061d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:35:06.759 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:35:06.757+0000 7f7b1061d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:35:06.761 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:35:06.757+0000 7f7b1061d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:35:07.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-16T07:35:07.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:35:07.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-16T07:35:07.207 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:35:07.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:35:07.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:35:07.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:35:07.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:35:07.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:35:07.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:35:07.332 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:35:07.329+0000 7f7b1061d880 -1 Falling back to public interface 2024-09-16T07:35:07.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:35:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:35:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:35:08.638 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:35:08.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:35:08.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:35:08.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:35:09.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:35:09.871 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:35:09.869+0000 7f7b1061d880 -1 osd.4 0 log_to_monitors true 2024-09-16T07:35:10.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:35:10.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:35:10.076 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:35:10.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:35:10.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:35:10.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:35:10.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:35:11.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:35:11.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:35:11.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:35:11.564 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:35:11.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:35:11.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:35:12.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:35:13.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:35:13.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:35:13.108 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:35:13.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:35:13.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:35:13.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:35:13.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:35:14.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:35:14.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:35:14.675 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:35:14.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:35:14.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:35:14.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:35:15.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:35:16.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:35:16.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:35:16.278 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:35:16.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:35:16.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:35:16.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:35:16.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:35:17.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:35:17.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:35:17.827 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:35:17.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:35:17.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:35:17.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:35:18.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:35:19.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:35:19.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:35:19.265 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:35:19.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:35:19.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:35:19.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:35:19.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:35:20.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:35:20.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:35:20.879 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:35:20.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:35:20.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:35:20.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:35:21.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:35:22.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:35:22.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:35:22.354 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:35:22.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:35:22.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:35:22.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:35:22.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:35:23.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:35:23.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:35:23.812 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:35:23.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:35:23.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:35:23.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:35:24.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:35:25.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:35:25.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:35:25.477 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:35:25.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:35:25.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:35:25.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:35:26.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:35:27.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:35:27.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:35:27.293 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:35:27.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:35:27.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:35:27.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:35:28.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:35:29.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:35:29.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:35:29.509 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:35:29.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:35:29.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:35:29.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:35:29.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:35:30.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:35:30.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:35:30.997 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:35:30.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:35:30.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:35:30.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:35:31.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:35:32.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:35:32.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:35:32.775 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:35:32.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:35:32.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:35:32.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:35:33.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:35:34.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:35:34.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:35:34.600 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:35:34.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:35:34.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:35:34.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:35:35.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:35:36.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:35:36.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:35:36.239 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-16T07:35:36.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T07:35:36.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:35:36.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:35:37.064 INFO:tasks.workunit.client.0.smithi162.stdout:osd.4 up in weight 1 up_from 37 up_thru 39 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1732068193,v1:127.0.0.1:6835/1732068193] [v2:127.0.0.1:6836/1732068193,v1:127.0.0.1:6837/1732068193] exists,up d5ac2151-dfcc-4bef-ac20-82303ffee290 2024-09-16T07:35:37.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:35:37.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:35:37.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:35:37.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-16T07:35:37.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T07:35:37.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T07:35:37.066 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:35:37.066 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:35:37.066 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:35:37.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.0.asok 2024-09-16T07:35:37.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-16T07:35:37.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.0.asok log flush 2024-09-16T07:35:37.166 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:35:37.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-16T07:35:37.182 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:32:43.525+0000 7faf18640880 0 load: jerasure load: lrc 2024-09-16T07:35:37.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:520: TEST_ec_backfill_unfound: CEPH_ARGS='--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:35:37.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:522: TEST_ec_backfill_unfound: local poolname=pool-jerasure 2024-09-16T07:35:37.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:523: TEST_ec_backfill_unfound: create_erasure_coded_pool pool-jerasure 3 2 2024-09-16T07:35:37.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T07:35:37.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-16T07:35:37.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-16T07:35:37.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-16T07:35:37.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-16T07:35:37.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-16T07:35:37.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-16T07:35:37.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-16T07:35:37.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-16T07:35:38.918 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-16T07:35:38.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T07:35:39.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-16T07:35:39.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T07:35:39.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T07:35:39.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T07:35:39.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T07:35:39.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T07:35:39.934 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T07:35:39.934 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T07:35:39.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T07:35:39.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T07:35:39.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T07:35:40.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T07:35:40.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T07:35:40.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T07:35:40.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T07:35:40.133 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T07:35:40.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T07:35:40.622 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:35:40.622 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:35:40.622 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:35:40.622 INFO:tasks.workunit.client.0.smithi162.stderr:4' 2024-09-16T07:35:40.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T07:35:40.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:35:40.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T07:35:40.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803816 2024-09-16T07:35:40.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803816 2024-09-16T07:35:40.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803816' 2024-09-16T07:35:40.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:35:40.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T07:35:40.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607585 2024-09-16T07:35:40.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607585 2024-09-16T07:35:40.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803816 1-51539607585' 2024-09-16T07:35:40.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:35:40.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T07:35:41.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=77309411358 2024-09-16T07:35:41.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 77309411358 2024-09-16T07:35:41.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803816 1-51539607585 2-77309411358' 2024-09-16T07:35:41.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:35:41.134 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T07:35:41.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=115964117019 2024-09-16T07:35:41.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 115964117019 2024-09-16T07:35:41.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803816 1-51539607585 2-77309411358 3-115964117019' 2024-09-16T07:35:41.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:35:41.317 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T07:35:41.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=158913789965 2024-09-16T07:35:41.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 158913789965 2024-09-16T07:35:41.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803816 1-51539607585 2-77309411358 3-115964117019 4-158913789965' 2024-09-16T07:35:41.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:35:41.472 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803816 2024-09-16T07:35:41.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:35:41.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T07:35:41.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803816 2024-09-16T07:35:41.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:35:41.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803816 2024-09-16T07:35:41.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803816' 2024-09-16T07:35:41.478 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803816 2024-09-16T07:35:41.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:35:41.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803814 -lt 25769803816 2024-09-16T07:35:41.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T07:35:42.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T07:35:42.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:35:43.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803816 -lt 25769803816 2024-09-16T07:35:43.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:35:43.374 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607585 2024-09-16T07:35:43.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:35:43.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T07:35:43.378 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607585 2024-09-16T07:35:43.378 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:35:43.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607585 2024-09-16T07:35:43.380 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 51539607585 2024-09-16T07:35:43.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607585' 2024-09-16T07:35:43.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T07:35:43.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607585 -lt 51539607585 2024-09-16T07:35:43.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:35:43.857 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-77309411358 2024-09-16T07:35:43.857 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:35:43.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T07:35:43.860 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-77309411358 2024-09-16T07:35:43.860 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:35:43.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=77309411358 2024-09-16T07:35:43.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 77309411358' 2024-09-16T07:35:43.862 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 77309411358 2024-09-16T07:35:43.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T07:35:44.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 77309411358 -lt 77309411358 2024-09-16T07:35:44.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:35:44.281 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-115964117019 2024-09-16T07:35:44.281 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:35:44.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T07:35:44.284 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-115964117019 2024-09-16T07:35:44.284 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:35:44.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=115964117019 2024-09-16T07:35:44.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 115964117019' 2024-09-16T07:35:44.286 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 115964117019 2024-09-16T07:35:44.286 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T07:35:44.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 115964117020 -lt 115964117019 2024-09-16T07:35:44.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:35:44.844 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-158913789965 2024-09-16T07:35:44.844 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:35:44.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T07:35:44.844 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-158913789965 2024-09-16T07:35:44.844 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:35:44.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=158913789965 2024-09-16T07:35:44.845 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 158913789965 2024-09-16T07:35:44.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 158913789965' 2024-09-16T07:35:44.845 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T07:35:45.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 158913789965 -lt 158913789965 2024-09-16T07:35:45.200 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T07:35:45.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:35:45.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:35:45.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T07:35:45.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T07:35:45.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T07:35:45.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T07:35:45.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T07:35:45.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T07:35:45.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T07:35:45.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T07:35:46.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T07:35:46.166 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T07:35:46.167 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:35:46.167 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:35:46.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T07:35:46.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T07:35:46.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T07:35:46.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:525: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-09-16T07:35:47.134 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T07:35:47.134 INFO:tasks.workunit.client.0.smithi162.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T07:35:40.747133+0000 0'0 44:13 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-16T07:35:38.601014+0000 0'0 2024-09-16T07:35:38.601014+0000 0 0 periodic scrub scheduled @ 2024-09-17T10:02:44.786920+0000 0 0 2024-09-16T07:35:47.134 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T07:35:37.629944+0000 0'0 44:14 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-16T07:32:34.822302+0000 0'0 2024-09-16T07:32:34.822302+0000 0 0 periodic scrub scheduled @ 2024-09-17T17:25:14.282127+0000 0 0 2024-09-16T07:35:47.134 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T07:35:37.505291+0000 0'0 44:94 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-16T07:32:34.822302+0000 0'0 2024-09-16T07:32:34.822302+0000 0 0 periodic scrub scheduled @ 2024-09-17T18:26:31.815230+0000 0 0 2024-09-16T07:35:47.134 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T07:35:37.471091+0000 0'0 44:14 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-16T07:32:34.822302+0000 0'0 2024-09-16T07:32:34.822302+0000 0 0 periodic scrub scheduled @ 2024-09-17T15:01:13.786765+0000 0 0 2024-09-16T07:35:47.134 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T07:34:22.025558+0000 0'0 45:43 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T07:32:34.822302+0000 0'0 2024-09-16T07:32:34.822302+0000 0 0 periodic scrub scheduled @ 2024-09-17T15:44:09.084405+0000 0 0 2024-09-16T07:35:47.135 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-09-16T07:35:47.135 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T07:35:47.137 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-09-16T07:35:47.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:527: TEST_ec_backfill_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-16T07:35:47.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-16T07:35:47.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-16T07:35:47.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-16T07:35:47.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:35:47.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-16T07:35:47.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:35:47.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-16T07:35:47.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:35:47.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-16T07:35:47.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:35:47.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-16T07:35:47.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:47.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:528: TEST_ec_backfill_unfound: get_primary pool-jerasure myobject 2024-09-16T07:35:47.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=pool-jerasure 2024-09-16T07:35:47.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=myobject 2024-09-16T07:35:47.348 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map pool-jerasure myobject 2024-09-16T07:35:47.348 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T07:35:47.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:528: TEST_ec_backfill_unfound: local primary=3 2024-09-16T07:35:47.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T07:35:47.770 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: get_osds pool-jerasure myobject 2024-09-16T07:35:47.770 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T07:35:47.770 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-16T07:35:47.771 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-16T07:35:47.772 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:35:48.213 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T07:35:48.213 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:35:48.213 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:35:48.213 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T07:35:48.213 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T07:35:48.213 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2 2024-09-16T07:35:48.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: local -a initial_osds 2024-09-16T07:35:48.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:531: TEST_ec_backfill_unfound: local last_osd=2 2024-09-16T07:35:48.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:532: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-16T07:35:48.214 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:35:48.215 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:35:48.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:35:48.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:35:48.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:35:48.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:35:48.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:533: TEST_ec_backfill_unfound: ceph osd down 2 2024-09-16T07:35:49.486 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 is already down. 2024-09-16T07:35:49.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:534: TEST_ec_backfill_unfound: ceph osd out 2 2024-09-16T07:35:50.286 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 is already out. 2024-09-16T07:35:50.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:536: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-09-16T07:35:50.701 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T07:35:50.701 INFO:tasks.workunit.client.0.smithi162.stdout:2.0 1 0 0 0 0 4096 0 0 1 0 1 active+clean 2024-09-16T07:35:40.747133+0000 45'1 45:14 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-16T07:35:38.601014+0000 0'0 2024-09-16T07:35:38.601014+0000 0 0 periodic scrub scheduled @ 2024-09-17T10:02:44.786920+0000 0 0 2024-09-16T07:35:50.701 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T07:35:37.629944+0000 0'0 44:15 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-16T07:32:34.822302+0000 0'0 2024-09-16T07:32:34.822302+0000 0 0 periodic scrub scheduled @ 2024-09-17T17:25:14.282127+0000 0 0 2024-09-16T07:35:50.701 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T07:35:37.505291+0000 0'0 46:96 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-16T07:32:34.822302+0000 0'0 2024-09-16T07:32:34.822302+0000 0 0 periodic scrub scheduled @ 2024-09-17T18:26:31.815230+0000 0 0 2024-09-16T07:35:50.701 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T07:35:37.471091+0000 0'0 44:15 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-16T07:32:34.822302+0000 0'0 2024-09-16T07:32:34.822302+0000 0 0 periodic scrub scheduled @ 2024-09-17T15:01:13.786765+0000 0 0 2024-09-16T07:35:50.701 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 peering 2024-09-16T07:35:48.623481+0000 0'0 46:46 [1,0] 1 [1,0] 1 0'0 2024-09-16T07:32:34.822302+0000 0'0 2024-09-16T07:32:34.822302+0000 0 0 periodic scrub scheduled @ 2024-09-17T17:35:58.189739+0000 0 0 2024-09-16T07:35:50.701 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-09-16T07:35:50.701 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T07:35:50.704 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-09-16T07:35:50.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:538: TEST_ec_backfill_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2024-09-16T07:35:50.715 INFO:tasks.workunit.client.0.smithi162.stderr:4+0 records in 2024-09-16T07:35:50.715 INFO:tasks.workunit.client.0.smithi162.stderr:4+0 records out 2024-09-16T07:35:50.715 INFO:tasks.workunit.client.0.smithi162.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000292764 s, 14.0 MB/s 2024-09-16T07:35:50.716 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: seq 1 300 2024-09-16T07:35:50.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:50.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:51.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:51.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:51.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:51.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:51.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:51.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:52.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:52.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:52.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:52.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:52.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:52.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:52.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:52.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:53.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:53.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:53.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:53.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:53.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:53.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:54.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:54.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:54.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:54.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:54.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:54.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:54.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:54.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:55.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:55.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:55.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:55.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:55.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:55.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:55.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:55.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:56.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:56.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:56.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:56.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:56.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:56.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:56.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:56.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:57.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:57.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:57.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:57.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:57.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:57.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:57.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:57.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:58.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:58.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:58.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:58.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:58.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:58.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:58.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:58.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:59.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:59.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:59.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:59.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:59.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:59.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2024-09-16T07:35:59.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:35:59.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:00.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:00.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:00.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:00.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:00.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:00.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:00.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:00.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:01.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:01.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:01.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:01.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:01.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:01.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:01.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:01.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:02.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:02.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:02.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:02.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:02.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:02.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:02.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:02.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:03.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:03.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:03.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:03.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:03.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:03.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:03.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:03.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:04.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:04.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:04.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:04.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:04.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:04.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:05.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:05.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:05.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:05.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:05.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:05.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:05.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:05.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:06.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:06.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:06.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:06.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:06.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:06.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:06.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:06.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:07.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:07.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:07.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:07.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:07.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:07.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:07.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:07.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:08.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:08.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:08.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:08.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:08.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:08.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:08.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:08.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:09.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:09.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:09.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:09.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:09.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:09.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:09.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:09.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:10.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:10.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:10.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:10.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:10.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:10.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:10.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:10.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:11.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:11.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:11.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:11.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:11.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:11.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:12.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:12.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:12.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:12.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:12.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:12.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:12.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:12.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:13.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:13.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:13.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:13.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:13.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:13.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:13.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:13.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:14.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:14.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:14.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:14.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:14.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:14.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:14.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:14.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:15.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:15.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:15.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:15.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:15.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:15.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:15.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:15.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:16.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:16.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:16.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:16.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:16.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:16.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:17.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:17.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj101 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:17.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:17.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj102 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:17.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:17.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj103 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:17.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:17.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj104 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:18.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:18.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj105 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:18.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:18.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj106 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:18.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:18.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj107 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:18.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:18.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj108 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:19.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:19.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj109 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:19.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:19.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj110 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:19.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:19.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj111 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:19.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:19.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj112 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj113 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:20.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:20.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj114 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:20.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:20.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj115 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:20.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:20.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj116 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:21.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:21.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj117 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:21.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:21.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj118 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:21.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:21.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj119 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:21.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:21.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj120 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:22.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:22.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj121 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:22.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:22.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj122 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:22.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:22.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj123 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:22.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:22.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj124 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:23.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:23.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj125 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:23.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:23.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj126 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:23.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:23.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj127 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:23.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:23.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj128 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:24.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:24.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj129 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:24.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:24.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj130 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:24.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:24.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj131 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:24.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:24.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj132 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:25.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:25.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj133 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:25.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:25.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj134 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:25.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:25.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj135 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:25.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:25.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj136 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:26.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:26.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj137 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:26.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:26.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj138 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:26.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:26.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj139 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:27.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:27.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj140 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:27.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:27.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj141 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:27.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:27.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj142 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:27.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:27.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj143 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:28.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:28.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj144 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:28.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:28.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj145 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:28.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:28.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj146 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:28.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:28.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj147 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:29.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:29.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj148 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:29.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:29.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj149 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:29.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:29.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj150 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:29.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:29.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj151 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:30.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:30.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj152 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:30.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:30.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj153 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:30.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:30.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj154 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:30.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:30.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj155 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:31.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:31.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj156 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:31.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:31.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj157 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:31.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:31.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj158 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:31.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:31.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj159 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:32.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:32.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj160 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:32.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:32.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj161 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:32.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:32.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj162 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:33.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:33.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj163 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:33.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:33.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj164 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:33.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:33.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj165 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:33.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:33.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj166 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:34.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:34.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj167 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:34.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:34.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj168 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:34.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:34.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj169 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:34.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:34.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj170 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:35.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:35.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj171 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:35.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:35.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj172 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:35.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:35.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj173 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:35.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:35.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj174 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:36.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:36.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj175 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:36.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:36.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj176 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:36.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:36.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj177 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:36.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:36.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj178 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:37.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:37.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj179 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:37.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:37.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj180 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:37.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:37.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj181 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:37.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:37.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj182 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:38.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:38.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj183 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:38.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:38.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj184 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:38.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:38.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj185 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:38.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:38.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj186 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:39.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:39.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj187 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:39.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:39.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj188 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:39.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:39.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj189 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:40.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:40.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj190 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:40.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:40.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj191 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:40.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:40.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj192 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:40.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:40.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj193 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:41.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:41.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj194 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:41.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:41.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj195 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:41.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:41.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj196 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:41.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:41.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj197 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:42.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:42.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj198 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:42.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:42.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj199 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:42.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:42.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj200 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:43.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:43.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj201 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:43.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:43.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj202 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:43.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:43.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj203 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:43.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:43.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj204 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:44.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:44.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj205 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:44.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:44.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj206 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:44.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:44.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj207 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:45.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:45.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj208 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:45.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:45.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj209 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:45.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:45.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj210 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:45.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:45.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj211 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:46.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:46.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj212 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:46.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:46.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj213 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:46.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:46.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj214 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:46.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:46.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj215 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:47.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:47.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj216 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:47.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:47.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj217 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:47.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:47.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj218 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:48.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:48.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj219 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:48.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:48.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj220 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:48.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:48.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj221 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:49.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:49.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj222 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:49.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:49.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj223 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:49.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:49.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj224 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:49.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:49.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj225 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:49.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:49.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj226 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:50.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:50.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj227 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:50.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:50.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj228 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:50.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:50.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj229 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:51.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:51.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj230 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:51.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:51.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj231 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:51.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:51.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj232 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:51.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:51.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj233 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:52.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:52.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj234 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:52.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:52.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj235 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:52.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:52.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj236 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:52.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:52.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj237 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:53.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:53.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj238 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:53.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:53.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj239 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:53.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:53.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj240 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:53.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:53.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj241 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:54.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:54.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj242 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:54.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:54.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj243 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:54.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:54.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj244 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:55.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:55.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj245 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:55.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:55.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj246 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:55.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:55.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj247 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:55.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:55.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj248 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:56.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:56.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj249 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:56.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:56.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj250 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:56.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:56.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj251 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:56.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:56.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj252 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:57.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:57.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj253 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:57.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:57.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj254 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:57.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:57.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj255 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:57.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:57.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj256 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:58.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:58.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj257 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:58.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:58.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj258 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:58.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:58.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj259 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:59.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:59.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj260 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:59.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:59.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj261 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:59.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:59.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj262 td/test-erasure-eio/ORIGINAL 2024-09-16T07:36:59.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:36:59.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj263 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:00.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:00.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj264 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:00.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:00.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj265 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:00.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:00.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj266 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:00.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:00.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj267 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:01.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:01.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj268 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:01.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:01.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj269 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:01.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:01.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj270 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:01.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:01.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj271 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:02.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:02.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj272 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:02.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:02.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj273 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:02.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:02.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj274 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:02.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:02.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj275 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:03.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:03.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj276 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:03.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:03.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj277 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:03.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:03.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj278 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:04.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:04.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj279 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:04.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:04.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj280 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:04.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:04.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj281 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:04.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:04.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj282 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:05.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:05.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj283 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:05.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:05.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj284 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:05.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:05.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj285 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:05.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:05.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj286 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:06.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:06.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj287 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:06.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:06.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj288 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:06.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:06.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj289 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:06.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:06.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj290 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:07.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:07.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj291 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:07.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:07.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj292 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:07.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:07.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj293 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:07.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:07.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj294 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:08.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:08.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj295 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:08.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:08.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj296 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:08.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:08.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj297 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:08.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:08.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj298 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:09.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:09.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj299 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:09.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:37:09.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj300 td/test-erasure-eio/ORIGINAL 2024-09-16T07:37:09.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:544: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 0 2024-09-16T07:37:09.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=ec 2024-09-16T07:37:09.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T07:37:09.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T07:37:09.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T07:37:09.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-jerasure 2024-09-16T07:37:09.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T07:37:09.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj250 2024-09-16T07:37:09.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T07:37:09.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/test-erasure-eio 2024-09-16T07:37:09.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T07:37:09.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T07:37:09.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T07:37:09.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T07:37:09.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-jerasure obj250 2024-09-16T07:37:09.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T07:37:09.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-09-16T07:37:09.790 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj250 2024-09-16T07:37:09.790 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:37:10.295 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T07:37:10.295 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:37:10.296 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:37:10.296 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T07:37:10.296 INFO:tasks.workunit.client.0.smithi162.stderr:2147483647' 2024-09-16T07:37:10.296 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-09-16T07:37:10.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T07:37:10.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T07:37:10.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' ec '!=' ec ']' 2024-09-16T07:37:10.298 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/test-erasure-eio/3/type 2024-09-16T07:37:10.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T07:37:10.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T07:37:10.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T07:37:10.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T07:37:10.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T07:37:10.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T07:37:10.300 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T07:37:10.300 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T07:37:10.301 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T07:37:10.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T07:37:10.301 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:37:10.301 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:37:10.301 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:37:10.301 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.3.asok 2024-09-16T07:37:10.302 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79228/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T07:37:10.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T07:37:10.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T07:37:10.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T07:37:10.399 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T07:37:10.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T07:37:10.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T07:37:10.400 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:37:10.400 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:37:10.400 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:37:10.401 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.3.asok 2024-09-16T07:37:10.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T07:37:10.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.3.asok injectdataerr pool-jerasure obj250 0 2024-09-16T07:37:10.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:545: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 1 2024-09-16T07:37:10.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=ec 2024-09-16T07:37:10.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T07:37:10.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T07:37:10.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T07:37:10.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-jerasure 2024-09-16T07:37:10.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T07:37:10.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj250 2024-09-16T07:37:10.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T07:37:10.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/test-erasure-eio 2024-09-16T07:37:10.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T07:37:10.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=1 2024-09-16T07:37:10.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T07:37:10.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T07:37:10.504 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-jerasure obj250 2024-09-16T07:37:10.504 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T07:37:10.504 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-09-16T07:37:10.505 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj250 2024-09-16T07:37:10.505 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:37:11.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T07:37:11.009 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:37:11.009 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:37:11.009 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T07:37:11.009 INFO:tasks.workunit.client.0.smithi162.stderr:2147483647' 2024-09-16T07:37:11.010 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-09-16T07:37:11.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T07:37:11.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=1 2024-09-16T07:37:11.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' ec '!=' ec ']' 2024-09-16T07:37:11.010 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/test-erasure-eio/1/type 2024-09-16T07:37:11.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T07:37:11.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-16T07:37:11.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T07:37:11.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-16T07:37:11.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T07:37:11.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T07:37:11.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T07:37:11.014 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-16T07:37:11.014 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T07:37:11.014 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T07:37:11.015 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:37:11.015 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:37:11.015 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:37:11.015 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.1.asok 2024-09-16T07:37:11.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79228/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-16T07:37:11.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T07:37:11.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T07:37:11.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T07:37:11.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.1 2024-09-16T07:37:11.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T07:37:11.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T07:37:11.115 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:37:11.115 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:37:11.115 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:37:11.115 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.1.asok 2024-09-16T07:37:11.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T07:37:11.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.1.asok injectdataerr pool-jerasure obj250 1 2024-09-16T07:37:11.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:547: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2024-09-16T07:37:11.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-16T07:37:11.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T07:37:11.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T07:37:11.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T07:37:11.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-16T07:37:11.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:37:11.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:37:11.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:37:11.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:37:11.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-16T07:37:11.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-16T07:37:11.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T07:37:11.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T07:37:11.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:37:11.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T07:37:11.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:37:11.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:37:11.221 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:37:11.221 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:37:11.222 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:37:11.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:37:11.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:37:11.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:37:11.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:37:11.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:37:11.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:37:11.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:37:11.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:37:11.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:37:11.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T07:37:11.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T07:37:11.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-16T07:37:11.225 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-16T07:37:11.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T07:37:11.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:37:11.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-16T07:37:11.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T07:37:11.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T07:37:11.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T07:37:11.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T07:37:11.251 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:37:11.247+0000 7fed820c2880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:37:11.254 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:37:11.247+0000 7fed820c2880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:37:11.255 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:37:11.251+0000 7fed820c2880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:37:11.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T07:37:11.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:37:11.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T07:37:11.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:37:11.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:37:11.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:37:11.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:37:11.670 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:37:11.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:37:11.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:37:11.800 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:37:11.795+0000 7fed820c2880 -1 Falling back to public interface 2024-09-16T07:37:12.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:37:13.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:37:13.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:37:13.103 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:37:13.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:37:13.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:37:13.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:37:13.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:37:14.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:37:14.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:37:14.506 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:37:14.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:37:14.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:37:14.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:37:14.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:37:14.947 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:37:14.943+0000 7fed820c2880 -1 osd.2 45 log_to_monitors true 2024-09-16T07:37:15.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:37:15.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:37:15.941 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:37:15.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:37:15.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:37:15.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:37:16.369 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up out weight 0 up_from 53 up_thru 19 down_at 46 last_clean_interval [18,45) [v2:127.0.0.1:6818/3731286795,v1:127.0.0.1:6819/3731286795] [v2:127.0.0.1:6820/3731286795,v1:127.0.0.1:6821/3731286795] exists,up f6fd136e-d706-4bc6-a2ca-2b71afe2aa2d 2024-09-16T07:37:16.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:37:16.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:37:16.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:37:16.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:548: TEST_ec_backfill_unfound: ceph osd in 2 2024-09-16T07:37:17.148 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 is already in. 2024-09-16T07:37:17.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: sleep 15 2024-09-16T07:37:32.161 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: seq 1 240 2024-09-16T07:37:32.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-16T07:37:32.163 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-16T07:37:32.163 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-16T07:37:32.163 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-16T07:37:32.163 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-16T07:37:32.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-16T07:37:32.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:32.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:32.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-16T07:37:32.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-16T07:37:32.599 INFO:tasks.workunit.client.0.smithi162.stdout:active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:32.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:32.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-16T07:37:33.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-16T07:37:33.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-16T07:37:33.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-16T07:37:33.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-16T07:37:33.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-16T07:37:33.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-16T07:37:34.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:34.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:34.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-16T07:37:34.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-16T07:37:34.043 INFO:tasks.workunit.client.0.smithi162.stdout:active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:34.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:34.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-16T07:37:35.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-16T07:37:35.045 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-16T07:37:35.045 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-16T07:37:35.045 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-16T07:37:35.045 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-16T07:37:35.046 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-16T07:37:35.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:35.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:35.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-16T07:37:35.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-16T07:37:35.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:35.488 INFO:tasks.workunit.client.0.smithi162.stdout:active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:35.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-16T07:37:36.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-16T07:37:36.490 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-16T07:37:36.490 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-16T07:37:36.490 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-16T07:37:36.491 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-16T07:37:36.491 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-16T07:37:36.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:36.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:36.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-16T07:37:36.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-16T07:37:36.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:36.920 INFO:tasks.workunit.client.0.smithi162.stdout:active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:36.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-16T07:37:37.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-16T07:37:37.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-16T07:37:37.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-16T07:37:37.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-16T07:37:37.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-16T07:37:37.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-16T07:37:38.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:38.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:38.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-16T07:37:38.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-16T07:37:38.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:38.369 INFO:tasks.workunit.client.0.smithi162.stdout:active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:38.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-16T07:37:39.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-16T07:37:39.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-16T07:37:39.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-16T07:37:39.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-16T07:37:39.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-16T07:37:39.372 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-16T07:37:39.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:39.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:39.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-16T07:37:39.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-16T07:37:39.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:39.799 INFO:tasks.workunit.client.0.smithi162.stdout:active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:39.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-16T07:37:40.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-16T07:37:40.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-16T07:37:40.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-16T07:37:40.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-16T07:37:40.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-16T07:37:40.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-16T07:37:41.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:41.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:41.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-16T07:37:41.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-16T07:37:41.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:41.233 INFO:tasks.workunit.client.0.smithi162.stdout:active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:41.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-16T07:37:42.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-16T07:37:42.235 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-16T07:37:42.235 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-16T07:37:42.236 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-16T07:37:42.236 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-16T07:37:42.236 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-16T07:37:42.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:42.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:42.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-16T07:37:42.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-16T07:37:42.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:42.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-16T07:37:42.656 INFO:tasks.workunit.client.0.smithi162.stdout:active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:43.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-16T07:37:43.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-16T07:37:43.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-16T07:37:43.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-16T07:37:43.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-16T07:37:43.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-16T07:37:44.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:44.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:44.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-16T07:37:44.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-16T07:37:44.074 INFO:tasks.workunit.client.0.smithi162.stdout:active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:44.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-16T07:37:44.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-16T07:37:45.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-16T07:37:45.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-16T07:37:45.077 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-16T07:37:45.077 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-16T07:37:45.077 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-16T07:37:45.077 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-16T07:37:45.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+backfill_unfound+undersized+degraded+remapped 2024-09-16T07:37:45.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+backfill_unfound+undersized+degraded+remapped 2024-09-16T07:37:45.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-16T07:37:45.488 INFO:tasks.workunit.client.0.smithi162.stdout:active+backfill_unfound+undersized+degraded+remapped 2024-09-16T07:37:45.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 0 = 0 ']' 2024-09-16T07:37:45.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: break 2024-09-16T07:37:45.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:562: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-09-16T07:37:45.900 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T07:37:45.900 INFO:tasks.workunit.client.0.smithi162.stdout:2.0 301 1 28 0 1 1232896 0 0 101 200 101 active+backfill_unfound+undersized+degraded+remapped 2024-09-16T07:37:42.223720+0000 51'301 58:896 [3,1,4,0,2] 3 [3,1,4,0,NONE] 3 0'0 2024-09-16T07:35:38.601014+0000 0'0 2024-09-16T07:35:38.601014+0000 0 0 periodic scrub scheduled @ 2024-09-17T11:19:21.074726+0000 0 0 2024-09-16T07:37:45.900 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T07:37:18.386474+0000 0'0 57:52 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-16T07:32:34.822302+0000 0'0 2024-09-16T07:32:34.822302+0000 0 0 periodic scrub scheduled @ 2024-09-17T09:03:52.859640+0000 0 0 2024-09-16T07:37:45.901 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T07:35:37.505291+0000 0'0 58:106 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-16T07:32:34.822302+0000 0'0 2024-09-16T07:32:34.822302+0000 0 0 periodic scrub scheduled @ 2024-09-17T18:26:31.815230+0000 0 0 2024-09-16T07:37:45.901 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T07:35:37.471091+0000 0'0 57:28 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-16T07:32:34.822302+0000 0'0 2024-09-16T07:32:34.822302+0000 0 0 periodic scrub scheduled @ 2024-09-17T15:01:13.786765+0000 0 0 2024-09-16T07:37:45.901 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T07:37:18.386553+0000 0'0 57:79 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T07:32:34.822302+0000 0'0 2024-09-16T07:32:34.822302+0000 0 0 periodic scrub scheduled @ 2024-09-17T17:25:06.440850+0000 0 0 2024-09-16T07:37:45.901 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-09-16T07:37:45.901 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T07:37:45.903 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-09-16T07:37:45.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:563: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-16T07:37:45.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:37:45.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:37:45.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:37:45.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:37:45.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:37:46.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:37:46.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:564: TEST_ec_backfill_unfound: sleep 5 2024-09-16T07:37:51.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:566: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-09-16T07:37:51.630 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T07:37:51.630 INFO:tasks.workunit.client.0.smithi162.stdout:2.0 301 1 303 0 1 1232896 0 0 101 200 101 active+recovery_unfound+undersized+degraded 2024-09-16T07:37:47.209029+0000 51'301 60:909 [3,1,4,0,NONE] 3 [3,1,4,0,NONE] 3 0'0 2024-09-16T07:35:38.601014+0000 0'0 2024-09-16T07:35:38.601014+0000 0 0 periodic scrub scheduled @ 2024-09-17T11:47:48.149072+0000 0 0 2024-09-16T07:37:51.631 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T07:37:18.386474+0000 0'0 58:53 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-16T07:32:34.822302+0000 0'0 2024-09-16T07:32:34.822302+0000 0 0 periodic scrub scheduled @ 2024-09-17T09:03:52.859640+0000 0 0 2024-09-16T07:37:51.631 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T07:35:37.505291+0000 0'0 59:108 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-16T07:32:34.822302+0000 0'0 2024-09-16T07:32:34.822302+0000 0 0 periodic scrub scheduled @ 2024-09-17T18:26:31.815230+0000 0 0 2024-09-16T07:37:51.631 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T07:35:37.471091+0000 0'0 58:29 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-16T07:32:34.822302+0000 0'0 2024-09-16T07:32:34.822302+0000 0 0 periodic scrub scheduled @ 2024-09-17T15:01:13.786765+0000 0 0 2024-09-16T07:37:51.631 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-09-16T07:37:47.081769+0000 0'0 60:88 [1,0] 1 [1,0] 1 0'0 2024-09-16T07:32:34.822302+0000 0'0 2024-09-16T07:32:34.822302+0000 0 0 periodic scrub scheduled @ 2024-09-17T08:40:48.140737+0000 0 0 2024-09-16T07:37:51.631 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-09-16T07:37:51.631 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T07:37:51.634 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-09-16T07:37:51.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:567: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-16T07:37:51.820 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-09-16T07:37:51.821 INFO:tasks.workunit.client.0.smithi162.stdout: "num_missing": 1, 2024-09-16T07:37:51.821 INFO:tasks.workunit.client.0.smithi162.stdout: "num_unfound": 1, 2024-09-16T07:37:51.821 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": [ 2024-09-16T07:37:51.821 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T07:37:51.821 INFO:tasks.workunit.client.0.smithi162.stdout: "oid": { 2024-09-16T07:37:51.821 INFO:tasks.workunit.client.0.smithi162.stdout: "oid": "obj250", 2024-09-16T07:37:51.821 INFO:tasks.workunit.client.0.smithi162.stdout: "key": "", 2024-09-16T07:37:51.821 INFO:tasks.workunit.client.0.smithi162.stdout: "snapid": -2, 2024-09-16T07:37:51.821 INFO:tasks.workunit.client.0.smithi162.stdout: "hash": 2249616407, 2024-09-16T07:37:51.821 INFO:tasks.workunit.client.0.smithi162.stdout: "max": 0, 2024-09-16T07:37:51.821 INFO:tasks.workunit.client.0.smithi162.stdout: "pool": 2, 2024-09-16T07:37:51.821 INFO:tasks.workunit.client.0.smithi162.stdout: "namespace": "" 2024-09-16T07:37:51.822 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T07:37:51.822 INFO:tasks.workunit.client.0.smithi162.stdout: "need": "51'251", 2024-09-16T07:37:51.822 INFO:tasks.workunit.client.0.smithi162.stdout: "have": "0'0", 2024-09-16T07:37:51.822 INFO:tasks.workunit.client.0.smithi162.stdout: "flags": "none", 2024-09-16T07:37:51.822 INFO:tasks.workunit.client.0.smithi162.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-16T07:37:51.822 INFO:tasks.workunit.client.0.smithi162.stdout: "locations": [ 2024-09-16T07:37:51.822 INFO:tasks.workunit.client.0.smithi162.stdout: "0(3)", 2024-09-16T07:37:51.822 INFO:tasks.workunit.client.0.smithi162.stdout: "4(2)" 2024-09-16T07:37:51.822 INFO:tasks.workunit.client.0.smithi162.stdout: ] 2024-09-16T07:37:51.822 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T07:37:51.822 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T07:37:51.822 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "NotRecovering", 2024-09-16T07:37:51.822 INFO:tasks.workunit.client.0.smithi162.stdout: "available_might_have_unfound": true, 2024-09-16T07:37:51.822 INFO:tasks.workunit.client.0.smithi162.stdout: "might_have_unfound": [ 2024-09-16T07:37:51.823 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T07:37:51.823 INFO:tasks.workunit.client.0.smithi162.stdout: "osd": "2(4)", 2024-09-16T07:37:51.823 INFO:tasks.workunit.client.0.smithi162.stdout: "status": "osd is down" 2024-09-16T07:37:51.823 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T07:37:51.823 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T07:37:51.823 INFO:tasks.workunit.client.0.smithi162.stdout: "more": false 2024-09-16T07:37:51.823 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-09-16T07:37:51.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:568: TEST_ec_backfill_unfound: ceph pg 2.0 query 2024-09-16T07:37:51.988 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-09-16T07:37:51.989 INFO:tasks.workunit.client.0.smithi162.stdout: "snap_trimq": "[]", 2024-09-16T07:37:51.989 INFO:tasks.workunit.client.0.smithi162.stdout: "snap_trimq_len": 0, 2024-09-16T07:37:51.989 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "active+recovery_unfound+undersized+degraded", 2024-09-16T07:37:51.989 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch": 60, 2024-09-16T07:37:51.989 INFO:tasks.workunit.client.0.smithi162.stdout: "up": [ 2024-09-16T07:37:51.989 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-16T07:37:51.989 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-16T07:37:51.989 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-16T07:37:51.989 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-16T07:37:51.989 INFO:tasks.workunit.client.0.smithi162.stdout: 2147483647 2024-09-16T07:37:51.989 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T07:37:51.990 INFO:tasks.workunit.client.0.smithi162.stdout: "acting": [ 2024-09-16T07:37:51.990 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-16T07:37:51.990 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-16T07:37:51.990 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-16T07:37:51.990 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-16T07:37:51.990 INFO:tasks.workunit.client.0.smithi162.stdout: 2147483647 2024-09-16T07:37:51.990 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T07:37:51.990 INFO:tasks.workunit.client.0.smithi162.stdout: "acting_recovery_backfill": [ 2024-09-16T07:37:51.990 INFO:tasks.workunit.client.0.smithi162.stdout: "0(3)", 2024-09-16T07:37:51.990 INFO:tasks.workunit.client.0.smithi162.stdout: "1(1)", 2024-09-16T07:37:51.990 INFO:tasks.workunit.client.0.smithi162.stdout: "3(0)", 2024-09-16T07:37:51.991 INFO:tasks.workunit.client.0.smithi162.stdout: "4(2)" 2024-09-16T07:37:51.991 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T07:37:51.991 INFO:tasks.workunit.client.0.smithi162.stdout: "info": { 2024-09-16T07:37:51.991 INFO:tasks.workunit.client.0.smithi162.stdout: "pgid": "2.0s0", 2024-09-16T07:37:51.991 INFO:tasks.workunit.client.0.smithi162.stdout: "last_update": "51'301", 2024-09-16T07:37:51.991 INFO:tasks.workunit.client.0.smithi162.stdout: "last_complete": "51'250", 2024-09-16T07:37:51.991 INFO:tasks.workunit.client.0.smithi162.stdout: "log_tail": "51'200", 2024-09-16T07:37:51.991 INFO:tasks.workunit.client.0.smithi162.stdout: "last_user_version": 301, 2024-09-16T07:37:51.991 INFO:tasks.workunit.client.0.smithi162.stdout: "last_backfill": "MAX", 2024-09-16T07:37:51.991 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [], 2024-09-16T07:37:51.992 INFO:tasks.workunit.client.0.smithi162.stdout: "history": { 2024-09-16T07:37:51.992 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_created": 43, 2024-09-16T07:37:51.992 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_pool_created": 43, 2024-09-16T07:37:51.992 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 60, 2024-09-16T07:37:51.992 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_started": 59, 2024-09-16T07:37:51.992 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 44, 2024-09-16T07:37:51.992 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_clean": 43, 2024-09-16T07:37:51.992 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_split": 0, 2024-09-16T07:37:51.992 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_marked_full": 0, 2024-09-16T07:37:51.992 INFO:tasks.workunit.client.0.smithi162.stdout: "same_up_since": 59, 2024-09-16T07:37:51.992 INFO:tasks.workunit.client.0.smithi162.stdout: "same_interval_since": 59, 2024-09-16T07:37:51.992 INFO:tasks.workunit.client.0.smithi162.stdout: "same_primary_since": 43, 2024-09-16T07:37:51.992 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-16T07:37:51.993 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-16T07:35:38.601014+0000", 2024-09-16T07:37:51.993 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-16T07:37:51.993 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-16T07:35:38.601014+0000", 2024-09-16T07:37:51.993 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-16T07:35:38.601014+0000", 2024-09-16T07:37:51.993 INFO:tasks.workunit.client.0.smithi162.stdout: "prior_readable_until_ub": 0 2024-09-16T07:37:51.993 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T07:37:51.993 INFO:tasks.workunit.client.0.smithi162.stdout: "stats": { 2024-09-16T07:37:51.993 INFO:tasks.workunit.client.0.smithi162.stdout: "version": "51'301", 2024-09-16T07:37:51.993 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_seq": 910, 2024-09-16T07:37:51.993 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_epoch": 60, 2024-09-16T07:37:51.993 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "active+recovery_unfound+undersized+degraded", 2024-09-16T07:37:51.993 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fresh": "2024-09-16T07:37:47.209029+0000", 2024-09-16T07:37:51.993 INFO:tasks.workunit.client.0.smithi162.stdout: "last_change": "2024-09-16T07:37:47.209029+0000", 2024-09-16T07:37:51.994 INFO:tasks.workunit.client.0.smithi162.stdout: "last_active": "2024-09-16T07:37:47.209029+0000", 2024-09-16T07:37:51.994 INFO:tasks.workunit.client.0.smithi162.stdout: "last_peered": "2024-09-16T07:37:47.209029+0000", 2024-09-16T07:37:51.994 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean": "2024-09-16T07:35:47.340639+0000", 2024-09-16T07:37:51.994 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_active": "2024-09-16T07:37:47.180228+0000", 2024-09-16T07:37:51.994 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_peered": "2024-09-16T07:37:47.180228+0000", 2024-09-16T07:37:51.994 INFO:tasks.workunit.client.0.smithi162.stdout: "last_unstale": "2024-09-16T07:37:47.209029+0000", 2024-09-16T07:37:51.994 INFO:tasks.workunit.client.0.smithi162.stdout: "last_undegraded": "2024-09-16T07:37:46.872250+0000", 2024-09-16T07:37:51.994 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fullsized": "2024-09-16T07:37:46.871509+0000", 2024-09-16T07:37:51.994 INFO:tasks.workunit.client.0.smithi162.stdout: "mapping_epoch": 59, 2024-09-16T07:37:51.994 INFO:tasks.workunit.client.0.smithi162.stdout: "log_start": "51'200", 2024-09-16T07:37:51.994 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_start": "51'200", 2024-09-16T07:37:51.994 INFO:tasks.workunit.client.0.smithi162.stdout: "created": 43, 2024-09-16T07:37:51.995 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 44, 2024-09-16T07:37:51.995 INFO:tasks.workunit.client.0.smithi162.stdout: "parent": "0.0", 2024-09-16T07:37:51.995 INFO:tasks.workunit.client.0.smithi162.stdout: "parent_split_bits": 0, 2024-09-16T07:37:51.995 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-16T07:37:51.995 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-16T07:35:38.601014+0000", 2024-09-16T07:37:51.995 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-16T07:37:51.995 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-16T07:35:38.601014+0000", 2024-09-16T07:37:51.995 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-16T07:35:38.601014+0000", 2024-09-16T07:37:51.995 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_scrubbed": 0, 2024-09-16T07:37:51.995 INFO:tasks.workunit.client.0.smithi162.stdout: "log_size": 101, 2024-09-16T07:37:51.995 INFO:tasks.workunit.client.0.smithi162.stdout: "log_dups_size": 200, 2024-09-16T07:37:51.995 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_size": 101, 2024-09-16T07:37:51.995 INFO:tasks.workunit.client.0.smithi162.stdout: "stats_invalid": false, 2024-09-16T07:37:51.995 INFO:tasks.workunit.client.0.smithi162.stdout: "dirty_stats_invalid": false, 2024-09-16T07:37:51.996 INFO:tasks.workunit.client.0.smithi162.stdout: "omap_stats_invalid": false, 2024-09-16T07:37:51.996 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_stats_invalid": false, 2024-09-16T07:37:51.996 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_bytes_stats_invalid": false, 2024-09-16T07:37:51.996 INFO:tasks.workunit.client.0.smithi162.stdout: "pin_stats_invalid": false, 2024-09-16T07:37:51.996 INFO:tasks.workunit.client.0.smithi162.stdout: "manifest_stats_invalid": false, 2024-09-16T07:37:51.996 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrimq_len": 0, 2024-09-16T07:37:51.996 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_duration": 0, 2024-09-16T07:37:51.996 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-17T11:47:48.149072+0000", 2024-09-16T07:37:51.996 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_duration": 0, 2024-09-16T07:37:51.996 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_trimmed": 0, 2024-09-16T07:37:51.996 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrim_duration": 0, 2024-09-16T07:37:51.996 INFO:tasks.workunit.client.0.smithi162.stdout: "stat_sum": { 2024-09-16T07:37:51.996 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes": 1232896, 2024-09-16T07:37:51.997 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects": 301, 2024-09-16T07:37:51.997 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_clones": 0, 2024-09-16T07:37:51.997 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_copies": 1505, 2024-09-16T07:37:51.997 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing_on_primary": 1, 2024-09-16T07:37:51.997 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing": 1, 2024-09-16T07:37:51.997 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_degraded": 303, 2024-09-16T07:37:51.997 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_misplaced": 0, 2024-09-16T07:37:51.997 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_unfound": 1, 2024-09-16T07:37:51.997 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_dirty": 301, 2024-09-16T07:37:51.997 INFO:tasks.workunit.client.0.smithi162.stdout: "num_whiteouts": 0, 2024-09-16T07:37:51.997 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read": 0, 2024-09-16T07:37:51.997 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read_kb": 0, 2024-09-16T07:37:51.997 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write": 301, 2024-09-16T07:37:51.998 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write_kb": 1204, 2024-09-16T07:37:51.998 INFO:tasks.workunit.client.0.smithi162.stdout: "num_scrub_errors": 0, 2024-09-16T07:37:51.998 INFO:tasks.workunit.client.0.smithi162.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T07:37:51.998 INFO:tasks.workunit.client.0.smithi162.stdout: "num_deep_scrub_errors": 0, 2024-09-16T07:37:51.998 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_recovered": 0, 2024-09-16T07:37:51.998 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_recovered": 0, 2024-09-16T07:37:51.998 INFO:tasks.workunit.client.0.smithi162.stdout: "num_keys_recovered": 0, 2024-09-16T07:37:51.998 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_omap": 0, 2024-09-16T07:37:51.998 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T07:37:51.998 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T07:37:51.998 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush": 0, 2024-09-16T07:37:51.998 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_kb": 0, 2024-09-16T07:37:51.998 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict": 0, 2024-09-16T07:37:51.998 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_kb": 0, 2024-09-16T07:37:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: "num_promote": 0, 2024-09-16T07:37:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_high": 0, 2024-09-16T07:37:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_low": 0, 2024-09-16T07:37:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_some": 0, 2024-09-16T07:37:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_full": 0, 2024-09-16T07:37:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_pinned": 0, 2024-09-16T07:37:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: "num_legacy_snapsets": 0, 2024-09-16T07:37:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: "num_large_omap_objects": 0, 2024-09-16T07:37:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_manifest": 0, 2024-09-16T07:37:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_bytes": 0, 2024-09-16T07:37:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_keys": 0, 2024-09-16T07:37:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_repaired": 0 2024-09-16T07:37:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T07:37:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: "up": [ 2024-09-16T07:37:52.000 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-16T07:37:52.000 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-16T07:37:52.000 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-16T07:37:52.000 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-16T07:37:52.000 INFO:tasks.workunit.client.0.smithi162.stdout: 2147483647 2024-09-16T07:37:52.000 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T07:37:52.000 INFO:tasks.workunit.client.0.smithi162.stdout: "acting": [ 2024-09-16T07:37:52.000 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-16T07:37:52.000 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-16T07:37:52.000 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-16T07:37:52.000 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-16T07:37:52.000 INFO:tasks.workunit.client.0.smithi162.stdout: 2147483647 2024-09-16T07:37:52.001 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T07:37:52.001 INFO:tasks.workunit.client.0.smithi162.stdout: "avail_no_missing": [ 2024-09-16T07:37:52.001 INFO:tasks.workunit.client.0.smithi162.stdout: "0(3)", 2024-09-16T07:37:52.001 INFO:tasks.workunit.client.0.smithi162.stdout: "4(2)" 2024-09-16T07:37:52.001 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T07:37:52.001 INFO:tasks.workunit.client.0.smithi162.stdout: "object_location_counts": [ 2024-09-16T07:37:52.001 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T07:37:52.001 INFO:tasks.workunit.client.0.smithi162.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-16T07:37:52.001 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": 300 2024-09-16T07:37:52.001 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T07:37:52.001 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T07:37:52.001 INFO:tasks.workunit.client.0.smithi162.stdout: "shards": "0(3),4(2)", 2024-09-16T07:37:52.002 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": 1 2024-09-16T07:37:52.002 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T07:37:52.002 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T07:37:52.002 INFO:tasks.workunit.client.0.smithi162.stdout: "blocked_by": [], 2024-09-16T07:37:52.002 INFO:tasks.workunit.client.0.smithi162.stdout: "up_primary": 3, 2024-09-16T07:37:52.002 INFO:tasks.workunit.client.0.smithi162.stdout: "acting_primary": 3, 2024-09-16T07:37:52.002 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [] 2024-09-16T07:37:52.002 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T07:37:52.002 INFO:tasks.workunit.client.0.smithi162.stdout: "empty": 0, 2024-09-16T07:37:52.002 INFO:tasks.workunit.client.0.smithi162.stdout: "dne": 0, 2024-09-16T07:37:52.002 INFO:tasks.workunit.client.0.smithi162.stdout: "incomplete": 0, 2024-09-16T07:37:52.002 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 60, 2024-09-16T07:37:52.002 INFO:tasks.workunit.client.0.smithi162.stdout: "hit_set_history": { 2024-09-16T07:37:52.002 INFO:tasks.workunit.client.0.smithi162.stdout: "current_last_update": "0'0", 2024-09-16T07:37:52.002 INFO:tasks.workunit.client.0.smithi162.stdout: "history": [] 2024-09-16T07:37:52.003 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T07:37:52.003 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T07:37:52.003 INFO:tasks.workunit.client.0.smithi162.stdout: "peer_info": [ 2024-09-16T07:37:52.003 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T07:37:52.003 INFO:tasks.workunit.client.0.smithi162.stdout: "peer": "0(3)", 2024-09-16T07:37:52.003 INFO:tasks.workunit.client.0.smithi162.stdout: "pgid": "2.0s3", 2024-09-16T07:37:52.003 INFO:tasks.workunit.client.0.smithi162.stdout: "last_update": "51'301", 2024-09-16T07:37:52.003 INFO:tasks.workunit.client.0.smithi162.stdout: "last_complete": "51'301", 2024-09-16T07:37:52.003 INFO:tasks.workunit.client.0.smithi162.stdout: "log_tail": "51'200", 2024-09-16T07:37:52.003 INFO:tasks.workunit.client.0.smithi162.stdout: "last_user_version": 301, 2024-09-16T07:37:52.003 INFO:tasks.workunit.client.0.smithi162.stdout: "last_backfill": "MAX", 2024-09-16T07:37:52.003 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [], 2024-09-16T07:37:52.003 INFO:tasks.workunit.client.0.smithi162.stdout: "history": { 2024-09-16T07:37:52.004 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_created": 43, 2024-09-16T07:37:52.004 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_pool_created": 43, 2024-09-16T07:37:52.004 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 60, 2024-09-16T07:37:52.004 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_started": 59, 2024-09-16T07:37:52.004 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 44, 2024-09-16T07:37:52.004 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_clean": 43, 2024-09-16T07:37:52.004 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_split": 0, 2024-09-16T07:37:52.004 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_marked_full": 0, 2024-09-16T07:37:52.004 INFO:tasks.workunit.client.0.smithi162.stdout: "same_up_since": 59, 2024-09-16T07:37:52.004 INFO:tasks.workunit.client.0.smithi162.stdout: "same_interval_since": 59, 2024-09-16T07:37:52.004 INFO:tasks.workunit.client.0.smithi162.stdout: "same_primary_since": 43, 2024-09-16T07:37:52.004 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-16T07:37:52.004 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-16T07:35:38.601014+0000", 2024-09-16T07:37:52.005 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-16T07:37:52.005 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-16T07:35:38.601014+0000", 2024-09-16T07:37:52.005 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-16T07:35:38.601014+0000", 2024-09-16T07:37:52.005 INFO:tasks.workunit.client.0.smithi162.stdout: "prior_readable_until_ub": 0 2024-09-16T07:37:52.005 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T07:37:52.005 INFO:tasks.workunit.client.0.smithi162.stdout: "stats": { 2024-09-16T07:37:52.005 INFO:tasks.workunit.client.0.smithi162.stdout: "version": "51'301", 2024-09-16T07:37:52.005 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_seq": 900, 2024-09-16T07:37:52.005 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_epoch": 59, 2024-09-16T07:37:52.005 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "active+undersized+degraded", 2024-09-16T07:37:52.005 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fresh": "2024-09-16T07:37:09.780803+0000", 2024-09-16T07:37:52.005 INFO:tasks.workunit.client.0.smithi162.stdout: "last_change": "2024-09-16T07:35:49.737830+0000", 2024-09-16T07:37:52.005 INFO:tasks.workunit.client.0.smithi162.stdout: "last_active": "2024-09-16T07:37:09.780803+0000", 2024-09-16T07:37:52.006 INFO:tasks.workunit.client.0.smithi162.stdout: "last_peered": "2024-09-16T07:37:09.780803+0000", 2024-09-16T07:37:52.006 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean": "2024-09-16T07:35:47.340639+0000", 2024-09-16T07:37:52.006 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_active": "2024-09-16T07:35:49.737830+0000", 2024-09-16T07:37:52.006 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_peered": "2024-09-16T07:35:49.737830+0000", 2024-09-16T07:37:52.006 INFO:tasks.workunit.client.0.smithi162.stdout: "last_unstale": "2024-09-16T07:37:09.780803+0000", 2024-09-16T07:37:52.006 INFO:tasks.workunit.client.0.smithi162.stdout: "last_undegraded": "2024-09-16T07:35:49.352397+0000", 2024-09-16T07:37:52.006 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fullsized": "2024-09-16T07:35:49.351965+0000", 2024-09-16T07:37:52.006 INFO:tasks.workunit.client.0.smithi162.stdout: "mapping_epoch": 59, 2024-09-16T07:37:52.006 INFO:tasks.workunit.client.0.smithi162.stdout: "log_start": "51'200", 2024-09-16T07:37:52.006 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_start": "51'200", 2024-09-16T07:37:52.006 INFO:tasks.workunit.client.0.smithi162.stdout: "created": 43, 2024-09-16T07:37:52.006 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 44, 2024-09-16T07:37:52.006 INFO:tasks.workunit.client.0.smithi162.stdout: "parent": "0.0", 2024-09-16T07:37:52.006 INFO:tasks.workunit.client.0.smithi162.stdout: "parent_split_bits": 0, 2024-09-16T07:37:52.007 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-16T07:37:52.007 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-16T07:35:38.601014+0000", 2024-09-16T07:37:52.007 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-16T07:37:52.007 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-16T07:35:38.601014+0000", 2024-09-16T07:37:52.007 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-16T07:35:38.601014+0000", 2024-09-16T07:37:52.007 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_scrubbed": 0, 2024-09-16T07:37:52.007 INFO:tasks.workunit.client.0.smithi162.stdout: "log_size": 101, 2024-09-16T07:37:52.007 INFO:tasks.workunit.client.0.smithi162.stdout: "log_dups_size": 200, 2024-09-16T07:37:52.007 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_size": 101, 2024-09-16T07:37:52.007 INFO:tasks.workunit.client.0.smithi162.stdout: "stats_invalid": false, 2024-09-16T07:37:52.007 INFO:tasks.workunit.client.0.smithi162.stdout: "dirty_stats_invalid": false, 2024-09-16T07:37:52.007 INFO:tasks.workunit.client.0.smithi162.stdout: "omap_stats_invalid": false, 2024-09-16T07:37:52.007 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_stats_invalid": false, 2024-09-16T07:37:52.007 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_bytes_stats_invalid": false, 2024-09-16T07:37:52.007 INFO:tasks.workunit.client.0.smithi162.stdout: "pin_stats_invalid": false, 2024-09-16T07:37:52.008 INFO:tasks.workunit.client.0.smithi162.stdout: "manifest_stats_invalid": false, 2024-09-16T07:37:52.008 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrimq_len": 0, 2024-09-16T07:37:52.008 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_duration": 0, 2024-09-16T07:37:52.008 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-17T08:15:53.786668+0000", 2024-09-16T07:37:52.008 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_duration": 0, 2024-09-16T07:37:52.008 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_trimmed": 0, 2024-09-16T07:37:52.008 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrim_duration": 0, 2024-09-16T07:37:52.008 INFO:tasks.workunit.client.0.smithi162.stdout: "stat_sum": { 2024-09-16T07:37:52.008 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes": 1232896, 2024-09-16T07:37:52.008 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects": 301, 2024-09-16T07:37:52.008 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_clones": 0, 2024-09-16T07:37:52.008 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_copies": 1505, 2024-09-16T07:37:52.008 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T07:37:52.008 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing": 0, 2024-09-16T07:37:52.009 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_degraded": 301, 2024-09-16T07:37:52.009 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_misplaced": 0, 2024-09-16T07:37:52.009 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_unfound": 0, 2024-09-16T07:37:52.009 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_dirty": 301, 2024-09-16T07:37:52.009 INFO:tasks.workunit.client.0.smithi162.stdout: "num_whiteouts": 0, 2024-09-16T07:37:52.009 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read": 0, 2024-09-16T07:37:52.009 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read_kb": 0, 2024-09-16T07:37:52.009 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write": 301, 2024-09-16T07:37:52.009 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write_kb": 1204, 2024-09-16T07:37:52.009 INFO:tasks.workunit.client.0.smithi162.stdout: "num_scrub_errors": 0, 2024-09-16T07:37:52.009 INFO:tasks.workunit.client.0.smithi162.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T07:37:52.009 INFO:tasks.workunit.client.0.smithi162.stdout: "num_deep_scrub_errors": 0, 2024-09-16T07:37:52.009 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_recovered": 0, 2024-09-16T07:37:52.010 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_recovered": 0, 2024-09-16T07:37:52.010 INFO:tasks.workunit.client.0.smithi162.stdout: "num_keys_recovered": 0, 2024-09-16T07:37:52.010 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_omap": 0, 2024-09-16T07:37:52.010 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T07:37:52.010 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T07:37:52.010 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush": 0, 2024-09-16T07:37:52.010 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_kb": 0, 2024-09-16T07:37:52.010 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict": 0, 2024-09-16T07:37:52.010 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_kb": 0, 2024-09-16T07:37:52.010 INFO:tasks.workunit.client.0.smithi162.stdout: "num_promote": 0, 2024-09-16T07:37:52.010 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_high": 0, 2024-09-16T07:37:52.010 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_low": 0, 2024-09-16T07:37:52.010 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_some": 0, 2024-09-16T07:37:52.010 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_full": 0, 2024-09-16T07:37:52.011 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_pinned": 0, 2024-09-16T07:37:52.011 INFO:tasks.workunit.client.0.smithi162.stdout: "num_legacy_snapsets": 0, 2024-09-16T07:37:52.011 INFO:tasks.workunit.client.0.smithi162.stdout: "num_large_omap_objects": 0, 2024-09-16T07:37:52.011 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_manifest": 0, 2024-09-16T07:37:52.011 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_bytes": 0, 2024-09-16T07:37:52.011 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_keys": 0, 2024-09-16T07:37:52.011 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_repaired": 0 2024-09-16T07:37:52.011 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T07:37:52.011 INFO:tasks.workunit.client.0.smithi162.stdout: "up": [ 2024-09-16T07:37:52.011 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-16T07:37:52.011 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-16T07:37:52.011 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-16T07:37:52.011 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-16T07:37:52.011 INFO:tasks.workunit.client.0.smithi162.stdout: 2147483647 2024-09-16T07:37:52.012 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T07:37:52.012 INFO:tasks.workunit.client.0.smithi162.stdout: "acting": [ 2024-09-16T07:37:52.012 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-16T07:37:52.012 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-16T07:37:52.012 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-16T07:37:52.012 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-16T07:37:52.012 INFO:tasks.workunit.client.0.smithi162.stdout: 2147483647 2024-09-16T07:37:52.012 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T07:37:52.012 INFO:tasks.workunit.client.0.smithi162.stdout: "avail_no_missing": [ 2024-09-16T07:37:52.012 INFO:tasks.workunit.client.0.smithi162.stdout: "3(0)", 2024-09-16T07:37:52.012 INFO:tasks.workunit.client.0.smithi162.stdout: "0(3)", 2024-09-16T07:37:52.012 INFO:tasks.workunit.client.0.smithi162.stdout: "1(1)", 2024-09-16T07:37:52.012 INFO:tasks.workunit.client.0.smithi162.stdout: "4(2)" 2024-09-16T07:37:52.012 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T07:37:52.013 INFO:tasks.workunit.client.0.smithi162.stdout: "object_location_counts": [ 2024-09-16T07:37:52.013 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T07:37:52.013 INFO:tasks.workunit.client.0.smithi162.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-16T07:37:52.013 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": 301 2024-09-16T07:37:52.013 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T07:37:52.013 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T07:37:52.013 INFO:tasks.workunit.client.0.smithi162.stdout: "blocked_by": [], 2024-09-16T07:37:52.013 INFO:tasks.workunit.client.0.smithi162.stdout: "up_primary": 3, 2024-09-16T07:37:52.013 INFO:tasks.workunit.client.0.smithi162.stdout: "acting_primary": 3, 2024-09-16T07:37:52.013 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [] 2024-09-16T07:37:52.013 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T07:37:52.013 INFO:tasks.workunit.client.0.smithi162.stdout: "empty": 0, 2024-09-16T07:37:52.013 INFO:tasks.workunit.client.0.smithi162.stdout: "dne": 0, 2024-09-16T07:37:52.014 INFO:tasks.workunit.client.0.smithi162.stdout: "incomplete": 0, 2024-09-16T07:37:52.014 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 60, 2024-09-16T07:37:52.014 INFO:tasks.workunit.client.0.smithi162.stdout: "hit_set_history": { 2024-09-16T07:37:52.014 INFO:tasks.workunit.client.0.smithi162.stdout: "current_last_update": "0'0", 2024-09-16T07:37:52.014 INFO:tasks.workunit.client.0.smithi162.stdout: "history": [] 2024-09-16T07:37:52.014 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T07:37:52.014 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T07:37:52.014 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T07:37:52.014 INFO:tasks.workunit.client.0.smithi162.stdout: "peer": "1(1)", 2024-09-16T07:37:52.014 INFO:tasks.workunit.client.0.smithi162.stdout: "pgid": "2.0s1", 2024-09-16T07:37:52.014 INFO:tasks.workunit.client.0.smithi162.stdout: "last_update": "51'301", 2024-09-16T07:37:52.014 INFO:tasks.workunit.client.0.smithi162.stdout: "last_complete": "51'301", 2024-09-16T07:37:52.014 INFO:tasks.workunit.client.0.smithi162.stdout: "log_tail": "51'200", 2024-09-16T07:37:52.014 INFO:tasks.workunit.client.0.smithi162.stdout: "last_user_version": 301, 2024-09-16T07:37:52.014 INFO:tasks.workunit.client.0.smithi162.stdout: "last_backfill": "MAX", 2024-09-16T07:37:52.015 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [], 2024-09-16T07:37:52.015 INFO:tasks.workunit.client.0.smithi162.stdout: "history": { 2024-09-16T07:37:52.015 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_created": 43, 2024-09-16T07:37:52.015 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_pool_created": 43, 2024-09-16T07:37:52.015 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 60, 2024-09-16T07:37:52.015 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_started": 59, 2024-09-16T07:37:52.015 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 44, 2024-09-16T07:37:52.015 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_clean": 43, 2024-09-16T07:37:52.015 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_split": 0, 2024-09-16T07:37:52.015 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_marked_full": 0, 2024-09-16T07:37:52.015 INFO:tasks.workunit.client.0.smithi162.stdout: "same_up_since": 59, 2024-09-16T07:37:52.015 INFO:tasks.workunit.client.0.smithi162.stdout: "same_interval_since": 59, 2024-09-16T07:37:52.015 INFO:tasks.workunit.client.0.smithi162.stdout: "same_primary_since": 43, 2024-09-16T07:37:52.015 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-16T07:37:52.015 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-16T07:35:38.601014+0000", 2024-09-16T07:37:52.015 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-16T07:37:52.016 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-16T07:35:38.601014+0000", 2024-09-16T07:37:52.016 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-16T07:35:38.601014+0000", 2024-09-16T07:37:52.016 INFO:tasks.workunit.client.0.smithi162.stdout: "prior_readable_until_ub": 0 2024-09-16T07:37:52.016 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T07:37:52.016 INFO:tasks.workunit.client.0.smithi162.stdout: "stats": { 2024-09-16T07:37:52.016 INFO:tasks.workunit.client.0.smithi162.stdout: "version": "51'301", 2024-09-16T07:37:52.016 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_seq": 327, 2024-09-16T07:37:52.016 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_epoch": 51, 2024-09-16T07:37:52.016 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "active+undersized+degraded", 2024-09-16T07:37:52.016 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fresh": "2024-09-16T07:37:09.780803+0000", 2024-09-16T07:37:52.016 INFO:tasks.workunit.client.0.smithi162.stdout: "last_change": "2024-09-16T07:35:49.737830+0000", 2024-09-16T07:37:52.016 INFO:tasks.workunit.client.0.smithi162.stdout: "last_active": "2024-09-16T07:37:09.780803+0000", 2024-09-16T07:37:52.016 INFO:tasks.workunit.client.0.smithi162.stdout: "last_peered": "2024-09-16T07:37:09.780803+0000", 2024-09-16T07:37:52.016 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean": "2024-09-16T07:35:47.340639+0000", 2024-09-16T07:37:52.016 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_active": "2024-09-16T07:35:49.737830+0000", 2024-09-16T07:37:52.017 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_peered": "2024-09-16T07:35:49.737830+0000", 2024-09-16T07:37:52.017 INFO:tasks.workunit.client.0.smithi162.stdout: "last_unstale": "2024-09-16T07:37:09.780803+0000", 2024-09-16T07:37:52.017 INFO:tasks.workunit.client.0.smithi162.stdout: "last_undegraded": "2024-09-16T07:35:49.352397+0000", 2024-09-16T07:37:52.017 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fullsized": "2024-09-16T07:35:49.351965+0000", 2024-09-16T07:37:52.017 INFO:tasks.workunit.client.0.smithi162.stdout: "mapping_epoch": 59, 2024-09-16T07:37:52.017 INFO:tasks.workunit.client.0.smithi162.stdout: "log_start": "51'200", 2024-09-16T07:37:52.017 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_start": "51'200", 2024-09-16T07:37:52.017 INFO:tasks.workunit.client.0.smithi162.stdout: "created": 43, 2024-09-16T07:37:52.017 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 44, 2024-09-16T07:37:52.017 INFO:tasks.workunit.client.0.smithi162.stdout: "parent": "0.0", 2024-09-16T07:37:52.017 INFO:tasks.workunit.client.0.smithi162.stdout: "parent_split_bits": 0, 2024-09-16T07:37:52.017 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-16T07:37:52.017 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-16T07:35:38.601014+0000", 2024-09-16T07:37:52.017 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-16T07:37:52.018 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-16T07:35:38.601014+0000", 2024-09-16T07:37:52.018 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-16T07:35:38.601014+0000", 2024-09-16T07:37:52.018 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_scrubbed": 0, 2024-09-16T07:37:52.018 INFO:tasks.workunit.client.0.smithi162.stdout: "log_size": 101, 2024-09-16T07:37:52.018 INFO:tasks.workunit.client.0.smithi162.stdout: "log_dups_size": 200, 2024-09-16T07:37:52.018 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_size": 101, 2024-09-16T07:37:52.018 INFO:tasks.workunit.client.0.smithi162.stdout: "stats_invalid": false, 2024-09-16T07:37:52.018 INFO:tasks.workunit.client.0.smithi162.stdout: "dirty_stats_invalid": false, 2024-09-16T07:37:52.018 INFO:tasks.workunit.client.0.smithi162.stdout: "omap_stats_invalid": false, 2024-09-16T07:37:52.018 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_stats_invalid": false, 2024-09-16T07:37:52.018 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_bytes_stats_invalid": false, 2024-09-16T07:37:52.018 INFO:tasks.workunit.client.0.smithi162.stdout: "pin_stats_invalid": false, 2024-09-16T07:37:52.018 INFO:tasks.workunit.client.0.smithi162.stdout: "manifest_stats_invalid": false, 2024-09-16T07:37:52.018 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrimq_len": 0, 2024-09-16T07:37:52.019 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_duration": 0, 2024-09-16T07:37:52.019 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-17T08:15:53.786668+0000", 2024-09-16T07:37:52.019 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_duration": 0, 2024-09-16T07:37:52.019 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_trimmed": 0, 2024-09-16T07:37:52.019 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrim_duration": 0, 2024-09-16T07:37:52.019 INFO:tasks.workunit.client.0.smithi162.stdout: "stat_sum": { 2024-09-16T07:37:52.019 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes": 1232896, 2024-09-16T07:37:52.019 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects": 301, 2024-09-16T07:37:52.019 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_clones": 0, 2024-09-16T07:37:52.019 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_copies": 1505, 2024-09-16T07:37:52.019 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T07:37:52.019 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing": 1, 2024-09-16T07:37:52.019 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_degraded": 301, 2024-09-16T07:37:52.019 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_misplaced": 0, 2024-09-16T07:37:52.019 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_unfound": 0, 2024-09-16T07:37:52.020 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_dirty": 301, 2024-09-16T07:37:52.020 INFO:tasks.workunit.client.0.smithi162.stdout: "num_whiteouts": 0, 2024-09-16T07:37:52.020 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read": 0, 2024-09-16T07:37:52.020 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read_kb": 0, 2024-09-16T07:37:52.020 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write": 301, 2024-09-16T07:37:52.020 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write_kb": 1204, 2024-09-16T07:37:52.020 INFO:tasks.workunit.client.0.smithi162.stdout: "num_scrub_errors": 0, 2024-09-16T07:37:52.020 INFO:tasks.workunit.client.0.smithi162.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T07:37:52.020 INFO:tasks.workunit.client.0.smithi162.stdout: "num_deep_scrub_errors": 0, 2024-09-16T07:37:52.020 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_recovered": 0, 2024-09-16T07:37:52.020 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_recovered": 0, 2024-09-16T07:37:52.020 INFO:tasks.workunit.client.0.smithi162.stdout: "num_keys_recovered": 0, 2024-09-16T07:37:52.020 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_omap": 0, 2024-09-16T07:37:52.020 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T07:37:52.021 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T07:37:52.021 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush": 0, 2024-09-16T07:37:52.021 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_kb": 0, 2024-09-16T07:37:52.021 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict": 0, 2024-09-16T07:37:52.021 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_kb": 0, 2024-09-16T07:37:52.021 INFO:tasks.workunit.client.0.smithi162.stdout: "num_promote": 0, 2024-09-16T07:37:52.021 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_high": 0, 2024-09-16T07:37:52.021 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_low": 0, 2024-09-16T07:37:52.021 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_some": 0, 2024-09-16T07:37:52.021 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_full": 0, 2024-09-16T07:37:52.021 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_pinned": 0, 2024-09-16T07:37:52.021 INFO:tasks.workunit.client.0.smithi162.stdout: "num_legacy_snapsets": 0, 2024-09-16T07:37:52.022 INFO:tasks.workunit.client.0.smithi162.stdout: "num_large_omap_objects": 0, 2024-09-16T07:37:52.022 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_manifest": 0, 2024-09-16T07:37:52.022 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_bytes": 0, 2024-09-16T07:37:52.022 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_keys": 0, 2024-09-16T07:37:52.022 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_repaired": 0 2024-09-16T07:37:52.022 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T07:37:52.022 INFO:tasks.workunit.client.0.smithi162.stdout: "up": [ 2024-09-16T07:37:52.022 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-16T07:37:52.022 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-16T07:37:52.022 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-16T07:37:52.022 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-16T07:37:52.022 INFO:tasks.workunit.client.0.smithi162.stdout: 2147483647 2024-09-16T07:37:52.022 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T07:37:52.022 INFO:tasks.workunit.client.0.smithi162.stdout: "acting": [ 2024-09-16T07:37:52.022 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-16T07:37:52.023 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-16T07:37:52.023 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-16T07:37:52.023 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-16T07:37:52.023 INFO:tasks.workunit.client.0.smithi162.stdout: 2147483647 2024-09-16T07:37:52.023 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T07:37:52.023 INFO:tasks.workunit.client.0.smithi162.stdout: "avail_no_missing": [ 2024-09-16T07:37:52.023 INFO:tasks.workunit.client.0.smithi162.stdout: "3(0)", 2024-09-16T07:37:52.023 INFO:tasks.workunit.client.0.smithi162.stdout: "0(3)", 2024-09-16T07:37:52.023 INFO:tasks.workunit.client.0.smithi162.stdout: "1(1)", 2024-09-16T07:37:52.023 INFO:tasks.workunit.client.0.smithi162.stdout: "4(2)" 2024-09-16T07:37:52.023 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T07:37:52.023 INFO:tasks.workunit.client.0.smithi162.stdout: "object_location_counts": [ 2024-09-16T07:37:52.023 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T07:37:52.023 INFO:tasks.workunit.client.0.smithi162.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-16T07:37:52.024 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": 301 2024-09-16T07:37:52.024 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T07:37:52.024 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T07:37:52.024 INFO:tasks.workunit.client.0.smithi162.stdout: "blocked_by": [], 2024-09-16T07:37:52.024 INFO:tasks.workunit.client.0.smithi162.stdout: "up_primary": 3, 2024-09-16T07:37:52.024 INFO:tasks.workunit.client.0.smithi162.stdout: "acting_primary": 3, 2024-09-16T07:37:52.024 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [] 2024-09-16T07:37:52.024 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T07:37:52.024 INFO:tasks.workunit.client.0.smithi162.stdout: "empty": 0, 2024-09-16T07:37:52.024 INFO:tasks.workunit.client.0.smithi162.stdout: "dne": 0, 2024-09-16T07:37:52.024 INFO:tasks.workunit.client.0.smithi162.stdout: "incomplete": 0, 2024-09-16T07:37:52.024 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 60, 2024-09-16T07:37:52.024 INFO:tasks.workunit.client.0.smithi162.stdout: "hit_set_history": { 2024-09-16T07:37:52.025 INFO:tasks.workunit.client.0.smithi162.stdout: "current_last_update": "0'0", 2024-09-16T07:37:52.025 INFO:tasks.workunit.client.0.smithi162.stdout: "history": [] 2024-09-16T07:37:52.025 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T07:37:52.025 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T07:37:52.025 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T07:37:52.025 INFO:tasks.workunit.client.0.smithi162.stdout: "peer": "4(2)", 2024-09-16T07:37:52.025 INFO:tasks.workunit.client.0.smithi162.stdout: "pgid": "2.0s2", 2024-09-16T07:37:52.025 INFO:tasks.workunit.client.0.smithi162.stdout: "last_update": "51'301", 2024-09-16T07:37:52.025 INFO:tasks.workunit.client.0.smithi162.stdout: "last_complete": "51'301", 2024-09-16T07:37:52.025 INFO:tasks.workunit.client.0.smithi162.stdout: "log_tail": "51'200", 2024-09-16T07:37:52.025 INFO:tasks.workunit.client.0.smithi162.stdout: "last_user_version": 301, 2024-09-16T07:37:52.025 INFO:tasks.workunit.client.0.smithi162.stdout: "last_backfill": "MAX", 2024-09-16T07:37:52.025 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [], 2024-09-16T07:37:52.026 INFO:tasks.workunit.client.0.smithi162.stdout: "history": { 2024-09-16T07:37:52.026 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_created": 43, 2024-09-16T07:37:52.026 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_pool_created": 43, 2024-09-16T07:37:52.026 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 60, 2024-09-16T07:37:52.026 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_started": 59, 2024-09-16T07:37:52.026 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 44, 2024-09-16T07:37:52.026 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_clean": 43, 2024-09-16T07:37:52.026 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_split": 0, 2024-09-16T07:37:52.026 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_marked_full": 0, 2024-09-16T07:37:52.026 INFO:tasks.workunit.client.0.smithi162.stdout: "same_up_since": 59, 2024-09-16T07:37:52.026 INFO:tasks.workunit.client.0.smithi162.stdout: "same_interval_since": 59, 2024-09-16T07:37:52.026 INFO:tasks.workunit.client.0.smithi162.stdout: "same_primary_since": 43, 2024-09-16T07:37:52.026 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-16T07:37:52.026 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-16T07:35:38.601014+0000", 2024-09-16T07:37:52.027 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-16T07:37:52.027 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-16T07:35:38.601014+0000", 2024-09-16T07:37:52.027 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-16T07:35:38.601014+0000", 2024-09-16T07:37:52.027 INFO:tasks.workunit.client.0.smithi162.stdout: "prior_readable_until_ub": 0 2024-09-16T07:37:52.027 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T07:37:52.027 INFO:tasks.workunit.client.0.smithi162.stdout: "stats": { 2024-09-16T07:37:52.027 INFO:tasks.workunit.client.0.smithi162.stdout: "version": "51'301", 2024-09-16T07:37:52.027 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_seq": 327, 2024-09-16T07:37:52.027 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_epoch": 51, 2024-09-16T07:37:52.027 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "active+undersized+degraded", 2024-09-16T07:37:52.027 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fresh": "2024-09-16T07:37:09.780803+0000", 2024-09-16T07:37:52.027 INFO:tasks.workunit.client.0.smithi162.stdout: "last_change": "2024-09-16T07:35:49.737830+0000", 2024-09-16T07:37:52.027 INFO:tasks.workunit.client.0.smithi162.stdout: "last_active": "2024-09-16T07:37:09.780803+0000", 2024-09-16T07:37:52.028 INFO:tasks.workunit.client.0.smithi162.stdout: "last_peered": "2024-09-16T07:37:09.780803+0000", 2024-09-16T07:37:52.028 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean": "2024-09-16T07:35:47.340639+0000", 2024-09-16T07:37:52.028 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_active": "2024-09-16T07:35:49.737830+0000", 2024-09-16T07:37:52.028 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_peered": "2024-09-16T07:35:49.737830+0000", 2024-09-16T07:37:52.028 INFO:tasks.workunit.client.0.smithi162.stdout: "last_unstale": "2024-09-16T07:37:09.780803+0000", 2024-09-16T07:37:52.028 INFO:tasks.workunit.client.0.smithi162.stdout: "last_undegraded": "2024-09-16T07:35:49.352397+0000", 2024-09-16T07:37:52.028 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fullsized": "2024-09-16T07:35:49.351965+0000", 2024-09-16T07:37:52.028 INFO:tasks.workunit.client.0.smithi162.stdout: "mapping_epoch": 59, 2024-09-16T07:37:52.028 INFO:tasks.workunit.client.0.smithi162.stdout: "log_start": "51'200", 2024-09-16T07:37:52.028 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_start": "51'200", 2024-09-16T07:37:52.028 INFO:tasks.workunit.client.0.smithi162.stdout: "created": 43, 2024-09-16T07:37:52.028 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 44, 2024-09-16T07:37:52.028 INFO:tasks.workunit.client.0.smithi162.stdout: "parent": "0.0", 2024-09-16T07:37:52.029 INFO:tasks.workunit.client.0.smithi162.stdout: "parent_split_bits": 0, 2024-09-16T07:37:52.029 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-16T07:37:52.029 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-16T07:35:38.601014+0000", 2024-09-16T07:37:52.029 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-16T07:37:52.029 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-16T07:35:38.601014+0000", 2024-09-16T07:37:52.029 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-16T07:35:38.601014+0000", 2024-09-16T07:37:52.029 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_scrubbed": 0, 2024-09-16T07:37:52.029 INFO:tasks.workunit.client.0.smithi162.stdout: "log_size": 101, 2024-09-16T07:37:52.029 INFO:tasks.workunit.client.0.smithi162.stdout: "log_dups_size": 200, 2024-09-16T07:37:52.029 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_size": 101, 2024-09-16T07:37:52.029 INFO:tasks.workunit.client.0.smithi162.stdout: "stats_invalid": false, 2024-09-16T07:37:52.029 INFO:tasks.workunit.client.0.smithi162.stdout: "dirty_stats_invalid": false, 2024-09-16T07:37:52.029 INFO:tasks.workunit.client.0.smithi162.stdout: "omap_stats_invalid": false, 2024-09-16T07:37:52.030 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_stats_invalid": false, 2024-09-16T07:37:52.030 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_bytes_stats_invalid": false, 2024-09-16T07:37:52.030 INFO:tasks.workunit.client.0.smithi162.stdout: "pin_stats_invalid": false, 2024-09-16T07:37:52.030 INFO:tasks.workunit.client.0.smithi162.stdout: "manifest_stats_invalid": false, 2024-09-16T07:37:52.030 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrimq_len": 0, 2024-09-16T07:37:52.030 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_duration": 0, 2024-09-16T07:37:52.030 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-17T08:15:53.786668+0000", 2024-09-16T07:37:52.030 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_duration": 0, 2024-09-16T07:37:52.030 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_trimmed": 0, 2024-09-16T07:37:52.030 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrim_duration": 0, 2024-09-16T07:37:52.030 INFO:tasks.workunit.client.0.smithi162.stdout: "stat_sum": { 2024-09-16T07:37:52.030 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes": 1232896, 2024-09-16T07:37:52.031 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects": 301, 2024-09-16T07:37:52.031 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_clones": 0, 2024-09-16T07:37:52.031 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_copies": 1505, 2024-09-16T07:37:52.031 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T07:37:52.031 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing": 0, 2024-09-16T07:37:52.031 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_degraded": 301, 2024-09-16T07:37:52.031 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_misplaced": 0, 2024-09-16T07:37:52.031 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_unfound": 0, 2024-09-16T07:37:52.031 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_dirty": 301, 2024-09-16T07:37:52.031 INFO:tasks.workunit.client.0.smithi162.stdout: "num_whiteouts": 0, 2024-09-16T07:37:52.031 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read": 0, 2024-09-16T07:37:52.031 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read_kb": 0, 2024-09-16T07:37:52.032 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write": 301, 2024-09-16T07:37:52.032 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write_kb": 1204, 2024-09-16T07:37:52.032 INFO:tasks.workunit.client.0.smithi162.stdout: "num_scrub_errors": 0, 2024-09-16T07:37:52.032 INFO:tasks.workunit.client.0.smithi162.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T07:37:52.032 INFO:tasks.workunit.client.0.smithi162.stdout: "num_deep_scrub_errors": 0, 2024-09-16T07:37:52.032 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_recovered": 0, 2024-09-16T07:37:52.032 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_recovered": 0, 2024-09-16T07:37:52.032 INFO:tasks.workunit.client.0.smithi162.stdout: "num_keys_recovered": 0, 2024-09-16T07:37:52.032 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_omap": 0, 2024-09-16T07:37:52.032 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T07:37:52.033 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T07:37:52.033 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush": 0, 2024-09-16T07:37:52.033 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_kb": 0, 2024-09-16T07:37:52.033 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict": 0, 2024-09-16T07:37:52.033 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_kb": 0, 2024-09-16T07:37:52.033 INFO:tasks.workunit.client.0.smithi162.stdout: "num_promote": 0, 2024-09-16T07:37:52.033 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_high": 0, 2024-09-16T07:37:52.033 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_low": 0, 2024-09-16T07:37:52.033 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_some": 0, 2024-09-16T07:37:52.033 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_full": 0, 2024-09-16T07:37:52.033 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_pinned": 0, 2024-09-16T07:37:52.033 INFO:tasks.workunit.client.0.smithi162.stdout: "num_legacy_snapsets": 0, 2024-09-16T07:37:52.033 INFO:tasks.workunit.client.0.smithi162.stdout: "num_large_omap_objects": 0, 2024-09-16T07:37:52.034 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_manifest": 0, 2024-09-16T07:37:52.034 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_bytes": 0, 2024-09-16T07:37:52.034 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_keys": 0, 2024-09-16T07:37:52.034 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_repaired": 0 2024-09-16T07:37:52.034 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T07:37:52.034 INFO:tasks.workunit.client.0.smithi162.stdout: "up": [ 2024-09-16T07:37:52.034 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-16T07:37:52.034 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-16T07:37:52.034 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-16T07:37:52.034 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-16T07:37:52.034 INFO:tasks.workunit.client.0.smithi162.stdout: 2147483647 2024-09-16T07:37:52.034 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T07:37:52.034 INFO:tasks.workunit.client.0.smithi162.stdout: "acting": [ 2024-09-16T07:37:52.034 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-16T07:37:52.034 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-16T07:37:52.035 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-16T07:37:52.035 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-16T07:37:52.035 INFO:tasks.workunit.client.0.smithi162.stdout: 2147483647 2024-09-16T07:37:52.035 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T07:37:52.035 INFO:tasks.workunit.client.0.smithi162.stdout: "avail_no_missing": [ 2024-09-16T07:37:52.035 INFO:tasks.workunit.client.0.smithi162.stdout: "3(0)", 2024-09-16T07:37:52.035 INFO:tasks.workunit.client.0.smithi162.stdout: "0(3)", 2024-09-16T07:37:52.035 INFO:tasks.workunit.client.0.smithi162.stdout: "1(1)", 2024-09-16T07:37:52.035 INFO:tasks.workunit.client.0.smithi162.stdout: "4(2)" 2024-09-16T07:37:52.035 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T07:37:52.035 INFO:tasks.workunit.client.0.smithi162.stdout: "object_location_counts": [ 2024-09-16T07:37:52.035 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T07:37:52.035 INFO:tasks.workunit.client.0.smithi162.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-16T07:37:52.035 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": 301 2024-09-16T07:37:52.035 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T07:37:52.036 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T07:37:52.036 INFO:tasks.workunit.client.0.smithi162.stdout: "blocked_by": [], 2024-09-16T07:37:52.036 INFO:tasks.workunit.client.0.smithi162.stdout: "up_primary": 3, 2024-09-16T07:37:52.036 INFO:tasks.workunit.client.0.smithi162.stdout: "acting_primary": 3, 2024-09-16T07:37:52.036 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [] 2024-09-16T07:37:52.036 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T07:37:52.036 INFO:tasks.workunit.client.0.smithi162.stdout: "empty": 0, 2024-09-16T07:37:52.036 INFO:tasks.workunit.client.0.smithi162.stdout: "dne": 0, 2024-09-16T07:37:52.036 INFO:tasks.workunit.client.0.smithi162.stdout: "incomplete": 0, 2024-09-16T07:37:52.036 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 60, 2024-09-16T07:37:52.036 INFO:tasks.workunit.client.0.smithi162.stdout: "hit_set_history": { 2024-09-16T07:37:52.036 INFO:tasks.workunit.client.0.smithi162.stdout: "current_last_update": "0'0", 2024-09-16T07:37:52.036 INFO:tasks.workunit.client.0.smithi162.stdout: "history": [] 2024-09-16T07:37:52.036 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T07:37:52.037 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T07:37:52.037 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T07:37:52.037 INFO:tasks.workunit.client.0.smithi162.stdout: "recovery_state": [ 2024-09-16T07:37:52.037 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T07:37:52.037 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "Started/Primary/Active", 2024-09-16T07:37:52.037 INFO:tasks.workunit.client.0.smithi162.stdout: "enter_time": "2024-09-16T07:37:46.871598+0000", 2024-09-16T07:37:52.037 INFO:tasks.workunit.client.0.smithi162.stdout: "might_have_unfound": [ 2024-09-16T07:37:52.037 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T07:37:52.037 INFO:tasks.workunit.client.0.smithi162.stdout: "osd": "0(3)", 2024-09-16T07:37:52.037 INFO:tasks.workunit.client.0.smithi162.stdout: "status": "already probed" 2024-09-16T07:37:52.037 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T07:37:52.037 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T07:37:52.037 INFO:tasks.workunit.client.0.smithi162.stdout: "osd": "1(1)", 2024-09-16T07:37:52.037 INFO:tasks.workunit.client.0.smithi162.stdout: "status": "already probed" 2024-09-16T07:37:52.038 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T07:37:52.038 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T07:37:52.038 INFO:tasks.workunit.client.0.smithi162.stdout: "osd": "2(4)", 2024-09-16T07:37:52.038 INFO:tasks.workunit.client.0.smithi162.stdout: "status": "osd is down" 2024-09-16T07:37:52.038 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T07:37:52.038 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T07:37:52.038 INFO:tasks.workunit.client.0.smithi162.stdout: "osd": "4(2)", 2024-09-16T07:37:52.038 INFO:tasks.workunit.client.0.smithi162.stdout: "status": "already probed" 2024-09-16T07:37:52.038 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T07:37:52.038 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T07:37:52.038 INFO:tasks.workunit.client.0.smithi162.stdout: "recovery_progress": { 2024-09-16T07:37:52.038 INFO:tasks.workunit.client.0.smithi162.stdout: "backfill_targets": [], 2024-09-16T07:37:52.038 INFO:tasks.workunit.client.0.smithi162.stdout: "waiting_on_backfill": [], 2024-09-16T07:37:52.038 INFO:tasks.workunit.client.0.smithi162.stdout: "last_backfill_started": "MIN", 2024-09-16T07:37:52.039 INFO:tasks.workunit.client.0.smithi162.stdout: "backfill_info": { 2024-09-16T07:37:52.039 INFO:tasks.workunit.client.0.smithi162.stdout: "begin": "MIN", 2024-09-16T07:37:52.039 INFO:tasks.workunit.client.0.smithi162.stdout: "end": "MIN", 2024-09-16T07:37:52.039 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": [] 2024-09-16T07:37:52.039 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T07:37:52.039 INFO:tasks.workunit.client.0.smithi162.stdout: "peer_backfill_info": [], 2024-09-16T07:37:52.039 INFO:tasks.workunit.client.0.smithi162.stdout: "backfills_in_flight": [], 2024-09-16T07:37:52.039 INFO:tasks.workunit.client.0.smithi162.stdout: "recovering": [], 2024-09-16T07:37:52.039 INFO:tasks.workunit.client.0.smithi162.stdout: "pg_backend": { 2024-09-16T07:37:52.039 INFO:tasks.workunit.client.0.smithi162.stdout: "recovery_ops": [], 2024-09-16T07:37:52.039 INFO:tasks.workunit.client.0.smithi162.stdout: "read_ops": [] 2024-09-16T07:37:52.039 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T07:37:52.039 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T07:37:52.039 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T07:37:52.039 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T07:37:52.040 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "Started", 2024-09-16T07:37:52.040 INFO:tasks.workunit.client.0.smithi162.stdout: "enter_time": "2024-09-16T07:37:46.287328+0000" 2024-09-16T07:37:52.040 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T07:37:52.040 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T07:37:52.040 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubber": { 2024-09-16T07:37:52.040 INFO:tasks.workunit.client.0.smithi162.stdout: "active": false, 2024-09-16T07:37:52.040 INFO:tasks.workunit.client.0.smithi162.stdout: "must_scrub": false, 2024-09-16T07:37:52.040 INFO:tasks.workunit.client.0.smithi162.stdout: "must_deep_scrub": false, 2024-09-16T07:37:52.040 INFO:tasks.workunit.client.0.smithi162.stdout: "must_repair": false, 2024-09-16T07:37:52.040 INFO:tasks.workunit.client.0.smithi162.stdout: "need_auto": false, 2024-09-16T07:37:52.040 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_reg_stamp": "2024-09-17T11:47:48.149072+0000", 2024-09-16T07:37:52.040 INFO:tasks.workunit.client.0.smithi162.stdout: "schedule": "scrub scheduled @ 2024-09-17T11:47:48.149" 2024-09-16T07:37:52.040 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T07:37:52.040 INFO:tasks.workunit.client.0.smithi162.stdout: "agent_state": {} 2024-09-16T07:37:52.041 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-09-16T07:37:52.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:570: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-16T07:37:52.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:570: TEST_ec_backfill_unfound: grep -q obj250 2024-09-16T07:37:52.154 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-16T07:37:52.154 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: jq .available_might_have_unfound 2024-09-16T07:37:52.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: check=true 2024-09-16T07:37:52.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: test true == true 2024-09-16T07:37:52.321 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-16T07:37:52.322 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].status' 2024-09-16T07:37:52.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: eval 'check="osd' is 'down"' 2024-09-16T07:37:52.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: check='osd is down' 2024-09-16T07:37:52.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:576: TEST_ec_backfill_unfound: test 'osd is down' == 'osd is down' 2024-09-16T07:37:52.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-16T07:37:52.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].osd' 2024-09-16T07:37:52.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: eval 'check="2(4)"' 2024-09-16T07:37:52.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: check='2(4)' 2024-09-16T07:37:52.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: test '2(4)' == '2(4)' 2024-09-16T07:37:52.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:581: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2024-09-16T07:37:52.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-16T07:37:52.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T07:37:52.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T07:37:52.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T07:37:52.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-16T07:37:52.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:37:52.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:37:52.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:37:52.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:37:52.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-16T07:37:52.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-16T07:37:52.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T07:37:52.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T07:37:52.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:37:52.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T07:37:52.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:37:52.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:37:52.634 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:37:52.635 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:37:52.635 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:37:52.636 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:37:52.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:37:52.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:37:52.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:37:52.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:37:52.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:37:52.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:37:52.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:37:52.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:37:52.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T07:37:52.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T07:37:52.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-16T07:37:52.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T07:37:52.638 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-16T07:37:52.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:37:52.640 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-16T07:37:52.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T07:37:52.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T07:37:52.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T07:37:52.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T07:37:52.672 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:37:52.670+0000 7fa83e564880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:37:52.675 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:37:52.674+0000 7fa83e564880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:37:52.677 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:37:52.674+0000 7fa83e564880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:37:53.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T07:37:53.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:37:53.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T07:37:53.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:37:53.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:37:53.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:37:53.100 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:37:53.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:37:53.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:37:53.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:37:53.220 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:37:53.218+0000 7fa83e564880 -1 Falling back to public interface 2024-09-16T07:37:53.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:37:54.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:37:54.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:37:54.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:37:54.540 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:37:54.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:37:54.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:37:54.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:37:55.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:37:55.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:37:55.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:37:55.965 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:37:55.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:37:55.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:37:56.151 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:37:56.150+0000 7fa83e564880 -1 osd.2 58 log_to_monitors true 2024-09-16T07:37:56.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:37:57.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:37:57.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:37:57.376 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:37:57.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:37:57.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:37:57.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:37:57.808 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 62 up_thru 19 down_at 59 last_clean_interval [53,58) [v2:127.0.0.1:6818/2238816552,v1:127.0.0.1:6819/2238816552] [v2:127.0.0.1:6820/2238816552,v1:127.0.0.1:6821/2238816552] exists,up f6fd136e-d706-4bc6-a2ca-2b71afe2aa2d 2024-09-16T07:37:57.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:37:57.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:37:57.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:37:57.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:584: TEST_ec_backfill_unfound: timeout 5 rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2024-09-16T07:38:02.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:585: TEST_ec_backfill_unfound: test 124 = 124 2024-09-16T07:38:02.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:587: TEST_ec_backfill_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-09-16T07:38:03.089 INFO:tasks.workunit.client.0.smithi162.stderr:pg has no unfound objects 2024-09-16T07:38:03.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:589: TEST_ec_backfill_unfound: wait_for_clean 2024-09-16T07:38:03.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T07:38:03.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T07:38:03.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T07:38:03.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T07:38:03.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T07:38:03.102 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T07:38:03.102 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T07:38:03.102 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T07:38:03.103 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T07:38:03.103 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T07:38:03.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T07:38:03.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T07:38:03.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T07:38:03.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T07:38:03.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T07:38:03.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T07:38:03.838 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:38:03.839 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:38:03.839 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:38:03.839 INFO:tasks.workunit.client.0.smithi162.stderr:4' 2024-09-16T07:38:03.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T07:38:03.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:38:03.839 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T07:38:04.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803846 2024-09-16T07:38:04.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803846 2024-09-16T07:38:04.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803846' 2024-09-16T07:38:04.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:38:04.001 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T07:38:04.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607615 2024-09-16T07:38:04.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607615 2024-09-16T07:38:04.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803846 1-51539607615' 2024-09-16T07:38:04.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:38:04.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T07:38:04.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=266287972356 2024-09-16T07:38:04.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 266287972356 2024-09-16T07:38:04.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803846 1-51539607615 2-266287972356' 2024-09-16T07:38:04.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:38:04.358 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T07:38:04.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=115964117050 2024-09-16T07:38:04.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 115964117050 2024-09-16T07:38:04.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803846 1-51539607615 2-266287972356 3-115964117050' 2024-09-16T07:38:04.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:38:04.523 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T07:38:04.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=158913789995 2024-09-16T07:38:04.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 158913789995 2024-09-16T07:38:04.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803846 1-51539607615 2-266287972356 3-115964117050 4-158913789995' 2024-09-16T07:38:04.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:38:04.678 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803846 2024-09-16T07:38:04.678 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:38:04.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T07:38:04.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803846 2024-09-16T07:38:04.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:38:04.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803846 2024-09-16T07:38:04.683 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803846 2024-09-16T07:38:04.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803846' 2024-09-16T07:38:04.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:38:05.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803846 -lt 25769803846 2024-09-16T07:38:05.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:38:05.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607615 2024-09-16T07:38:05.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:38:05.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T07:38:05.124 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607615 2024-09-16T07:38:05.124 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:38:05.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607615 2024-09-16T07:38:05.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607615' 2024-09-16T07:38:05.126 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 51539607615 2024-09-16T07:38:05.127 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T07:38:05.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607615 -lt 51539607615 2024-09-16T07:38:05.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:38:05.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-266287972356 2024-09-16T07:38:05.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:38:05.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T07:38:05.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-266287972356 2024-09-16T07:38:05.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:38:05.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=266287972356 2024-09-16T07:38:05.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 266287972356' 2024-09-16T07:38:05.598 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 266287972356 2024-09-16T07:38:05.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T07:38:06.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 266287972354 -lt 266287972356 2024-09-16T07:38:06.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T07:38:07.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T07:38:07.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T07:38:07.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 266287972356 -lt 266287972356 2024-09-16T07:38:07.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:38:07.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-115964117050 2024-09-16T07:38:07.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:38:07.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T07:38:07.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-115964117050 2024-09-16T07:38:07.477 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:38:07.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=115964117050 2024-09-16T07:38:07.479 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 115964117050 2024-09-16T07:38:07.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 115964117050' 2024-09-16T07:38:07.480 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T07:38:07.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 115964117050 -lt 115964117050 2024-09-16T07:38:07.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:38:07.930 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:38:07.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-158913789995 2024-09-16T07:38:07.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T07:38:07.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-158913789995 2024-09-16T07:38:07.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:38:07.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=158913789995 2024-09-16T07:38:07.935 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 158913789995 2024-09-16T07:38:07.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 158913789995' 2024-09-16T07:38:07.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T07:38:08.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 158913789996 -lt 158913789995 2024-09-16T07:38:08.453 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T07:38:08.453 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:38:08.453 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:38:09.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T07:38:09.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T07:38:09.051 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T07:38:09.051 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T07:38:09.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T07:38:09.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T07:38:09.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T07:38:09.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T07:38:09.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T07:38:09.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T07:38:09.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:38:09.474 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:38:10.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-09-16T07:38:10.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-09-16T07:38:10.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T07:38:10.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-09-16T07:38:10.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T07:38:10.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T07:38:10.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T07:38:10.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T07:38:10.196 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T07:38:10.196 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T07:38:10.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T07:38:10.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T07:38:10.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T07:38:10.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T07:38:10.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T07:38:10.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T07:38:10.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:38:10.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:38:11.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T07:38:11.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T07:38:11.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T07:38:11.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: seq 1 300 2024-09-16T07:38:11.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:11.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj1 = obj250 ']' 2024-09-16T07:38:11.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2024-09-16T07:38:11.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:11.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:11.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj2 = obj250 ']' 2024-09-16T07:38:11.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2024-09-16T07:38:11.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:11.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:11.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj3 = obj250 ']' 2024-09-16T07:38:11.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2024-09-16T07:38:11.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:11.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:11.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj4 = obj250 ']' 2024-09-16T07:38:11.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2024-09-16T07:38:11.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:11.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:11.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj5 = obj250 ']' 2024-09-16T07:38:11.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2024-09-16T07:38:11.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:11.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:11.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj6 = obj250 ']' 2024-09-16T07:38:11.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2024-09-16T07:38:11.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:11.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:11.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj7 = obj250 ']' 2024-09-16T07:38:11.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2024-09-16T07:38:11.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:11.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:11.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj8 = obj250 ']' 2024-09-16T07:38:11.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2024-09-16T07:38:11.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:11.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:11.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj9 = obj250 ']' 2024-09-16T07:38:11.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2024-09-16T07:38:11.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:11.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:11.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj10 = obj250 ']' 2024-09-16T07:38:11.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2024-09-16T07:38:11.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:11.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:11.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj11 = obj250 ']' 2024-09-16T07:38:11.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2024-09-16T07:38:11.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:11.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:11.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj12 = obj250 ']' 2024-09-16T07:38:11.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2024-09-16T07:38:11.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:11.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:11.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj13 = obj250 ']' 2024-09-16T07:38:11.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2024-09-16T07:38:11.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:11.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:11.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj14 = obj250 ']' 2024-09-16T07:38:11.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2024-09-16T07:38:11.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:11.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:11.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj15 = obj250 ']' 2024-09-16T07:38:11.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2024-09-16T07:38:11.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:11.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:11.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj16 = obj250 ']' 2024-09-16T07:38:11.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2024-09-16T07:38:11.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:11.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:11.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj17 = obj250 ']' 2024-09-16T07:38:11.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2024-09-16T07:38:11.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:11.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:11.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj18 = obj250 ']' 2024-09-16T07:38:11.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2024-09-16T07:38:12.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:12.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:12.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj19 = obj250 ']' 2024-09-16T07:38:12.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2024-09-16T07:38:12.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:12.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:12.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj20 = obj250 ']' 2024-09-16T07:38:12.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2024-09-16T07:38:12.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:12.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:12.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj21 = obj250 ']' 2024-09-16T07:38:12.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2024-09-16T07:38:12.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:12.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:12.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj22 = obj250 ']' 2024-09-16T07:38:12.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2024-09-16T07:38:12.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:12.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:12.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj23 = obj250 ']' 2024-09-16T07:38:12.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2024-09-16T07:38:12.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:12.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:12.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj24 = obj250 ']' 2024-09-16T07:38:12.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2024-09-16T07:38:12.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:12.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:12.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj25 = obj250 ']' 2024-09-16T07:38:12.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2024-09-16T07:38:12.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:12.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:12.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj26 = obj250 ']' 2024-09-16T07:38:12.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2024-09-16T07:38:12.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:12.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:12.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj27 = obj250 ']' 2024-09-16T07:38:12.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2024-09-16T07:38:12.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:12.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:12.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj28 = obj250 ']' 2024-09-16T07:38:12.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2024-09-16T07:38:12.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:12.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:12.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj29 = obj250 ']' 2024-09-16T07:38:12.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2024-09-16T07:38:12.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:12.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:12.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj30 = obj250 ']' 2024-09-16T07:38:12.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2024-09-16T07:38:12.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:12.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:12.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj31 = obj250 ']' 2024-09-16T07:38:12.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2024-09-16T07:38:12.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:12.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:12.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj32 = obj250 ']' 2024-09-16T07:38:12.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2024-09-16T07:38:12.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:12.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:12.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj33 = obj250 ']' 2024-09-16T07:38:12.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2024-09-16T07:38:12.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:12.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:12.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj34 = obj250 ']' 2024-09-16T07:38:12.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2024-09-16T07:38:12.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:12.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:12.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj35 = obj250 ']' 2024-09-16T07:38:12.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2024-09-16T07:38:13.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:13.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:13.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj36 = obj250 ']' 2024-09-16T07:38:13.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2024-09-16T07:38:13.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:13.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:13.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj37 = obj250 ']' 2024-09-16T07:38:13.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2024-09-16T07:38:13.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:13.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:13.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj38 = obj250 ']' 2024-09-16T07:38:13.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2024-09-16T07:38:13.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:13.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:13.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj39 = obj250 ']' 2024-09-16T07:38:13.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2024-09-16T07:38:13.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:13.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:13.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj40 = obj250 ']' 2024-09-16T07:38:13.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2024-09-16T07:38:13.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:13.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:13.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj41 = obj250 ']' 2024-09-16T07:38:13.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2024-09-16T07:38:13.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:13.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:13.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj42 = obj250 ']' 2024-09-16T07:38:13.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2024-09-16T07:38:13.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:13.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:13.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj43 = obj250 ']' 2024-09-16T07:38:13.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2024-09-16T07:38:13.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:13.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:13.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj44 = obj250 ']' 2024-09-16T07:38:13.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2024-09-16T07:38:13.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:13.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:13.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj45 = obj250 ']' 2024-09-16T07:38:13.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2024-09-16T07:38:13.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:13.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:13.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj46 = obj250 ']' 2024-09-16T07:38:13.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2024-09-16T07:38:13.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:13.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:13.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj47 = obj250 ']' 2024-09-16T07:38:13.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2024-09-16T07:38:13.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:13.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:13.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj48 = obj250 ']' 2024-09-16T07:38:13.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2024-09-16T07:38:13.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:13.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:13.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj49 = obj250 ']' 2024-09-16T07:38:13.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2024-09-16T07:38:13.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:13.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:13.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj50 = obj250 ']' 2024-09-16T07:38:13.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2024-09-16T07:38:13.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:13.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:13.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj51 = obj250 ']' 2024-09-16T07:38:13.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2024-09-16T07:38:13.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:13.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:13.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj52 = obj250 ']' 2024-09-16T07:38:13.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2024-09-16T07:38:13.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:13.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:13.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj53 = obj250 ']' 2024-09-16T07:38:13.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2024-09-16T07:38:13.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:13.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:13.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj54 = obj250 ']' 2024-09-16T07:38:13.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2024-09-16T07:38:13.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:13.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:13.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj55 = obj250 ']' 2024-09-16T07:38:13.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2024-09-16T07:38:14.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:14.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:14.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj56 = obj250 ']' 2024-09-16T07:38:14.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2024-09-16T07:38:14.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:14.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:14.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj57 = obj250 ']' 2024-09-16T07:38:14.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2024-09-16T07:38:14.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:14.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:14.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj58 = obj250 ']' 2024-09-16T07:38:14.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2024-09-16T07:38:14.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:14.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:14.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj59 = obj250 ']' 2024-09-16T07:38:14.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2024-09-16T07:38:14.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:14.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:14.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj60 = obj250 ']' 2024-09-16T07:38:14.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2024-09-16T07:38:14.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:14.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:14.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj61 = obj250 ']' 2024-09-16T07:38:14.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2024-09-16T07:38:14.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:14.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:14.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj62 = obj250 ']' 2024-09-16T07:38:14.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2024-09-16T07:38:14.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:14.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:14.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj63 = obj250 ']' 2024-09-16T07:38:14.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2024-09-16T07:38:14.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:14.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:14.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj64 = obj250 ']' 2024-09-16T07:38:14.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2024-09-16T07:38:14.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:14.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:14.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj65 = obj250 ']' 2024-09-16T07:38:14.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2024-09-16T07:38:14.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:14.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:14.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj66 = obj250 ']' 2024-09-16T07:38:14.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2024-09-16T07:38:14.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:14.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:14.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj67 = obj250 ']' 2024-09-16T07:38:14.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2024-09-16T07:38:14.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:14.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:14.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj68 = obj250 ']' 2024-09-16T07:38:14.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2024-09-16T07:38:14.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:14.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:14.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj69 = obj250 ']' 2024-09-16T07:38:14.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2024-09-16T07:38:14.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:14.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:14.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj70 = obj250 ']' 2024-09-16T07:38:14.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2024-09-16T07:38:14.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:14.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:14.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj71 = obj250 ']' 2024-09-16T07:38:14.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2024-09-16T07:38:14.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:14.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:14.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj72 = obj250 ']' 2024-09-16T07:38:14.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2024-09-16T07:38:14.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:14.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:14.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj73 = obj250 ']' 2024-09-16T07:38:14.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2024-09-16T07:38:15.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:15.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:15.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj74 = obj250 ']' 2024-09-16T07:38:15.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2024-09-16T07:38:15.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:15.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:15.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj75 = obj250 ']' 2024-09-16T07:38:15.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-09-16T07:38:15.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:15.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:15.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj76 = obj250 ']' 2024-09-16T07:38:15.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2024-09-16T07:38:15.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:15.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:15.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj77 = obj250 ']' 2024-09-16T07:38:15.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2024-09-16T07:38:15.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:15.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:15.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj78 = obj250 ']' 2024-09-16T07:38:15.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2024-09-16T07:38:15.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:15.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:15.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj79 = obj250 ']' 2024-09-16T07:38:15.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2024-09-16T07:38:15.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:15.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:15.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj80 = obj250 ']' 2024-09-16T07:38:15.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2024-09-16T07:38:15.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:15.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:15.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj81 = obj250 ']' 2024-09-16T07:38:15.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2024-09-16T07:38:15.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:15.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:15.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj82 = obj250 ']' 2024-09-16T07:38:15.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2024-09-16T07:38:15.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:15.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:15.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj83 = obj250 ']' 2024-09-16T07:38:15.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2024-09-16T07:38:15.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:15.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:15.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj84 = obj250 ']' 2024-09-16T07:38:15.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2024-09-16T07:38:15.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:15.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:15.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj85 = obj250 ']' 2024-09-16T07:38:15.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2024-09-16T07:38:15.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:15.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:15.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj86 = obj250 ']' 2024-09-16T07:38:15.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2024-09-16T07:38:15.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:15.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:15.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj87 = obj250 ']' 2024-09-16T07:38:15.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2024-09-16T07:38:15.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:15.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:15.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj88 = obj250 ']' 2024-09-16T07:38:15.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2024-09-16T07:38:15.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:15.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:15.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj89 = obj250 ']' 2024-09-16T07:38:15.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2024-09-16T07:38:15.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:15.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:15.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj90 = obj250 ']' 2024-09-16T07:38:15.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2024-09-16T07:38:15.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:15.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:15.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj91 = obj250 ']' 2024-09-16T07:38:15.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2024-09-16T07:38:15.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:15.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:15.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj92 = obj250 ']' 2024-09-16T07:38:15.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2024-09-16T07:38:15.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:15.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:15.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj93 = obj250 ']' 2024-09-16T07:38:15.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2024-09-16T07:38:16.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:16.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:16.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj94 = obj250 ']' 2024-09-16T07:38:16.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2024-09-16T07:38:16.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:16.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:16.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj95 = obj250 ']' 2024-09-16T07:38:16.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2024-09-16T07:38:16.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:16.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:16.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj96 = obj250 ']' 2024-09-16T07:38:16.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2024-09-16T07:38:16.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:16.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:16.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj97 = obj250 ']' 2024-09-16T07:38:16.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2024-09-16T07:38:16.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:16.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:16.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj98 = obj250 ']' 2024-09-16T07:38:16.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2024-09-16T07:38:16.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:16.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:16.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj99 = obj250 ']' 2024-09-16T07:38:16.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2024-09-16T07:38:16.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:16.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:16.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj100 = obj250 ']' 2024-09-16T07:38:16.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2024-09-16T07:38:16.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:16.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:16.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj101 = obj250 ']' 2024-09-16T07:38:16.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj101 td/test-erasure-eio/CHECK 2024-09-16T07:38:16.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:16.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:16.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj102 = obj250 ']' 2024-09-16T07:38:16.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj102 td/test-erasure-eio/CHECK 2024-09-16T07:38:16.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:16.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:16.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj103 = obj250 ']' 2024-09-16T07:38:16.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj103 td/test-erasure-eio/CHECK 2024-09-16T07:38:16.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:16.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:16.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj104 = obj250 ']' 2024-09-16T07:38:16.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj104 td/test-erasure-eio/CHECK 2024-09-16T07:38:16.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:16.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:16.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj105 = obj250 ']' 2024-09-16T07:38:16.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj105 td/test-erasure-eio/CHECK 2024-09-16T07:38:16.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:16.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:16.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj106 = obj250 ']' 2024-09-16T07:38:16.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj106 td/test-erasure-eio/CHECK 2024-09-16T07:38:16.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:16.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:16.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj107 = obj250 ']' 2024-09-16T07:38:16.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj107 td/test-erasure-eio/CHECK 2024-09-16T07:38:16.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:16.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:16.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj108 = obj250 ']' 2024-09-16T07:38:16.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj108 td/test-erasure-eio/CHECK 2024-09-16T07:38:16.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:16.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:16.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj109 = obj250 ']' 2024-09-16T07:38:16.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj109 td/test-erasure-eio/CHECK 2024-09-16T07:38:16.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:16.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:16.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj110 = obj250 ']' 2024-09-16T07:38:16.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj110 td/test-erasure-eio/CHECK 2024-09-16T07:38:16.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:16.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:16.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj111 = obj250 ']' 2024-09-16T07:38:16.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj111 td/test-erasure-eio/CHECK 2024-09-16T07:38:16.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:16.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:16.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj112 = obj250 ']' 2024-09-16T07:38:16.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj112 td/test-erasure-eio/CHECK 2024-09-16T07:38:17.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:17.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:17.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj113 = obj250 ']' 2024-09-16T07:38:17.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj113 td/test-erasure-eio/CHECK 2024-09-16T07:38:17.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:17.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:17.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj114 = obj250 ']' 2024-09-16T07:38:17.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj114 td/test-erasure-eio/CHECK 2024-09-16T07:38:17.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:17.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:17.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj115 = obj250 ']' 2024-09-16T07:38:17.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj115 td/test-erasure-eio/CHECK 2024-09-16T07:38:17.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:17.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:17.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj116 = obj250 ']' 2024-09-16T07:38:17.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj116 td/test-erasure-eio/CHECK 2024-09-16T07:38:17.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:17.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:17.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj117 = obj250 ']' 2024-09-16T07:38:17.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj117 td/test-erasure-eio/CHECK 2024-09-16T07:38:17.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:17.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:17.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj118 = obj250 ']' 2024-09-16T07:38:17.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj118 td/test-erasure-eio/CHECK 2024-09-16T07:38:17.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:17.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:17.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj119 = obj250 ']' 2024-09-16T07:38:17.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj119 td/test-erasure-eio/CHECK 2024-09-16T07:38:17.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:17.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:17.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj120 = obj250 ']' 2024-09-16T07:38:17.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj120 td/test-erasure-eio/CHECK 2024-09-16T07:38:17.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:17.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:17.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj121 = obj250 ']' 2024-09-16T07:38:17.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj121 td/test-erasure-eio/CHECK 2024-09-16T07:38:17.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:17.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:17.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj122 = obj250 ']' 2024-09-16T07:38:17.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj122 td/test-erasure-eio/CHECK 2024-09-16T07:38:17.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:17.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:17.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj123 = obj250 ']' 2024-09-16T07:38:17.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj123 td/test-erasure-eio/CHECK 2024-09-16T07:38:17.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:17.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:17.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj124 = obj250 ']' 2024-09-16T07:38:17.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj124 td/test-erasure-eio/CHECK 2024-09-16T07:38:17.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:17.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:17.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj125 = obj250 ']' 2024-09-16T07:38:17.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj125 td/test-erasure-eio/CHECK 2024-09-16T07:38:17.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:17.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:17.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj126 = obj250 ']' 2024-09-16T07:38:17.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj126 td/test-erasure-eio/CHECK 2024-09-16T07:38:17.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:17.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:17.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj127 = obj250 ']' 2024-09-16T07:38:17.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj127 td/test-erasure-eio/CHECK 2024-09-16T07:38:17.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:17.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:17.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj128 = obj250 ']' 2024-09-16T07:38:17.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj128 td/test-erasure-eio/CHECK 2024-09-16T07:38:17.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:17.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:17.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj129 = obj250 ']' 2024-09-16T07:38:17.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj129 td/test-erasure-eio/CHECK 2024-09-16T07:38:17.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:17.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:17.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj130 = obj250 ']' 2024-09-16T07:38:17.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj130 td/test-erasure-eio/CHECK 2024-09-16T07:38:17.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:17.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:17.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj131 = obj250 ']' 2024-09-16T07:38:17.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj131 td/test-erasure-eio/CHECK 2024-09-16T07:38:17.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:17.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:17.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj132 = obj250 ']' 2024-09-16T07:38:17.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj132 td/test-erasure-eio/CHECK 2024-09-16T07:38:17.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:17.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:17.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj133 = obj250 ']' 2024-09-16T07:38:18.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj133 td/test-erasure-eio/CHECK 2024-09-16T07:38:18.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:18.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:18.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj134 = obj250 ']' 2024-09-16T07:38:18.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj134 td/test-erasure-eio/CHECK 2024-09-16T07:38:18.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:18.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:18.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj135 = obj250 ']' 2024-09-16T07:38:18.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj135 td/test-erasure-eio/CHECK 2024-09-16T07:38:18.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:18.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:18.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj136 = obj250 ']' 2024-09-16T07:38:18.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj136 td/test-erasure-eio/CHECK 2024-09-16T07:38:18.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:18.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:18.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj137 = obj250 ']' 2024-09-16T07:38:18.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj137 td/test-erasure-eio/CHECK 2024-09-16T07:38:18.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:18.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:18.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj138 = obj250 ']' 2024-09-16T07:38:18.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj138 td/test-erasure-eio/CHECK 2024-09-16T07:38:18.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:18.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:18.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj139 = obj250 ']' 2024-09-16T07:38:18.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj139 td/test-erasure-eio/CHECK 2024-09-16T07:38:18.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:18.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:18.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj140 = obj250 ']' 2024-09-16T07:38:18.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj140 td/test-erasure-eio/CHECK 2024-09-16T07:38:18.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:18.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:18.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj141 = obj250 ']' 2024-09-16T07:38:18.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj141 td/test-erasure-eio/CHECK 2024-09-16T07:38:18.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:18.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:18.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj142 = obj250 ']' 2024-09-16T07:38:18.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj142 td/test-erasure-eio/CHECK 2024-09-16T07:38:18.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:18.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:18.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj143 = obj250 ']' 2024-09-16T07:38:18.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj143 td/test-erasure-eio/CHECK 2024-09-16T07:38:18.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:18.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:18.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj144 = obj250 ']' 2024-09-16T07:38:18.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj144 td/test-erasure-eio/CHECK 2024-09-16T07:38:18.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:18.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:18.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj145 = obj250 ']' 2024-09-16T07:38:18.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj145 td/test-erasure-eio/CHECK 2024-09-16T07:38:18.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:18.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:18.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj146 = obj250 ']' 2024-09-16T07:38:18.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj146 td/test-erasure-eio/CHECK 2024-09-16T07:38:18.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:18.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:18.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj147 = obj250 ']' 2024-09-16T07:38:18.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj147 td/test-erasure-eio/CHECK 2024-09-16T07:38:18.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:18.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:18.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj148 = obj250 ']' 2024-09-16T07:38:18.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj148 td/test-erasure-eio/CHECK 2024-09-16T07:38:18.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:18.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:18.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj149 = obj250 ']' 2024-09-16T07:38:18.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj149 td/test-erasure-eio/CHECK 2024-09-16T07:38:18.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:18.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:18.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj150 = obj250 ']' 2024-09-16T07:38:18.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj150 td/test-erasure-eio/CHECK 2024-09-16T07:38:18.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:18.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:18.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj151 = obj250 ']' 2024-09-16T07:38:18.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj151 td/test-erasure-eio/CHECK 2024-09-16T07:38:18.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:18.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:18.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj152 = obj250 ']' 2024-09-16T07:38:18.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj152 td/test-erasure-eio/CHECK 2024-09-16T07:38:19.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:19.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:19.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj153 = obj250 ']' 2024-09-16T07:38:19.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj153 td/test-erasure-eio/CHECK 2024-09-16T07:38:19.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:19.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:19.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj154 = obj250 ']' 2024-09-16T07:38:19.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj154 td/test-erasure-eio/CHECK 2024-09-16T07:38:19.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:19.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:19.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj155 = obj250 ']' 2024-09-16T07:38:19.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj155 td/test-erasure-eio/CHECK 2024-09-16T07:38:19.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:19.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:19.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj156 = obj250 ']' 2024-09-16T07:38:19.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj156 td/test-erasure-eio/CHECK 2024-09-16T07:38:19.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:19.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:19.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj157 = obj250 ']' 2024-09-16T07:38:19.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj157 td/test-erasure-eio/CHECK 2024-09-16T07:38:19.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:19.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:19.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj158 = obj250 ']' 2024-09-16T07:38:19.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj158 td/test-erasure-eio/CHECK 2024-09-16T07:38:19.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:19.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:19.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj159 = obj250 ']' 2024-09-16T07:38:19.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj159 td/test-erasure-eio/CHECK 2024-09-16T07:38:19.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:19.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:19.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj160 = obj250 ']' 2024-09-16T07:38:19.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj160 td/test-erasure-eio/CHECK 2024-09-16T07:38:19.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:19.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:19.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj161 = obj250 ']' 2024-09-16T07:38:19.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj161 td/test-erasure-eio/CHECK 2024-09-16T07:38:19.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:19.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:19.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj162 = obj250 ']' 2024-09-16T07:38:19.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj162 td/test-erasure-eio/CHECK 2024-09-16T07:38:19.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:19.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:19.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj163 = obj250 ']' 2024-09-16T07:38:19.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj163 td/test-erasure-eio/CHECK 2024-09-16T07:38:19.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:19.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:19.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj164 = obj250 ']' 2024-09-16T07:38:19.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj164 td/test-erasure-eio/CHECK 2024-09-16T07:38:19.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:19.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:19.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj165 = obj250 ']' 2024-09-16T07:38:19.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj165 td/test-erasure-eio/CHECK 2024-09-16T07:38:19.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:19.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:19.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj166 = obj250 ']' 2024-09-16T07:38:19.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj166 td/test-erasure-eio/CHECK 2024-09-16T07:38:19.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:19.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:19.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj167 = obj250 ']' 2024-09-16T07:38:19.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj167 td/test-erasure-eio/CHECK 2024-09-16T07:38:19.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:19.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:19.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj168 = obj250 ']' 2024-09-16T07:38:19.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj168 td/test-erasure-eio/CHECK 2024-09-16T07:38:19.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:19.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:19.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj169 = obj250 ']' 2024-09-16T07:38:19.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj169 td/test-erasure-eio/CHECK 2024-09-16T07:38:19.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:19.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:19.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj170 = obj250 ']' 2024-09-16T07:38:19.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj170 td/test-erasure-eio/CHECK 2024-09-16T07:38:19.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:19.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:19.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj171 = obj250 ']' 2024-09-16T07:38:19.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj171 td/test-erasure-eio/CHECK 2024-09-16T07:38:19.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:19.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:19.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj172 = obj250 ']' 2024-09-16T07:38:19.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj172 td/test-erasure-eio/CHECK 2024-09-16T07:38:20.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:20.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:20.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj173 = obj250 ']' 2024-09-16T07:38:20.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj173 td/test-erasure-eio/CHECK 2024-09-16T07:38:20.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:20.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:20.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj174 = obj250 ']' 2024-09-16T07:38:20.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj174 td/test-erasure-eio/CHECK 2024-09-16T07:38:20.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:20.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:20.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj175 = obj250 ']' 2024-09-16T07:38:20.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj175 td/test-erasure-eio/CHECK 2024-09-16T07:38:20.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:20.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:20.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj176 = obj250 ']' 2024-09-16T07:38:20.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj176 td/test-erasure-eio/CHECK 2024-09-16T07:38:20.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:20.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:20.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj177 = obj250 ']' 2024-09-16T07:38:20.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj177 td/test-erasure-eio/CHECK 2024-09-16T07:38:20.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:20.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:20.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj178 = obj250 ']' 2024-09-16T07:38:20.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj178 td/test-erasure-eio/CHECK 2024-09-16T07:38:20.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:20.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:20.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj179 = obj250 ']' 2024-09-16T07:38:20.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj179 td/test-erasure-eio/CHECK 2024-09-16T07:38:20.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:20.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:20.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj180 = obj250 ']' 2024-09-16T07:38:20.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj180 td/test-erasure-eio/CHECK 2024-09-16T07:38:20.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:20.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:20.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj181 = obj250 ']' 2024-09-16T07:38:20.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj181 td/test-erasure-eio/CHECK 2024-09-16T07:38:20.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:20.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:20.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj182 = obj250 ']' 2024-09-16T07:38:20.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj182 td/test-erasure-eio/CHECK 2024-09-16T07:38:20.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:20.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:20.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj183 = obj250 ']' 2024-09-16T07:38:20.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj183 td/test-erasure-eio/CHECK 2024-09-16T07:38:20.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:20.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:20.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj184 = obj250 ']' 2024-09-16T07:38:20.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj184 td/test-erasure-eio/CHECK 2024-09-16T07:38:20.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:20.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:20.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj185 = obj250 ']' 2024-09-16T07:38:20.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj185 td/test-erasure-eio/CHECK 2024-09-16T07:38:20.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:20.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:20.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj186 = obj250 ']' 2024-09-16T07:38:20.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj186 td/test-erasure-eio/CHECK 2024-09-16T07:38:20.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:20.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:20.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj187 = obj250 ']' 2024-09-16T07:38:20.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj187 td/test-erasure-eio/CHECK 2024-09-16T07:38:20.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:20.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:20.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj188 = obj250 ']' 2024-09-16T07:38:20.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj188 td/test-erasure-eio/CHECK 2024-09-16T07:38:20.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:20.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:20.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj189 = obj250 ']' 2024-09-16T07:38:20.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj189 td/test-erasure-eio/CHECK 2024-09-16T07:38:20.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:20.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:20.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj190 = obj250 ']' 2024-09-16T07:38:20.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj190 td/test-erasure-eio/CHECK 2024-09-16T07:38:20.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:20.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:20.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj191 = obj250 ']' 2024-09-16T07:38:20.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj191 td/test-erasure-eio/CHECK 2024-09-16T07:38:21.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:21.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:21.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj192 = obj250 ']' 2024-09-16T07:38:21.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj192 td/test-erasure-eio/CHECK 2024-09-16T07:38:21.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:21.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:21.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj193 = obj250 ']' 2024-09-16T07:38:21.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj193 td/test-erasure-eio/CHECK 2024-09-16T07:38:21.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:21.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:21.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj194 = obj250 ']' 2024-09-16T07:38:21.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj194 td/test-erasure-eio/CHECK 2024-09-16T07:38:21.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:21.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:21.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj195 = obj250 ']' 2024-09-16T07:38:21.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj195 td/test-erasure-eio/CHECK 2024-09-16T07:38:21.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:21.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:21.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj196 = obj250 ']' 2024-09-16T07:38:21.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj196 td/test-erasure-eio/CHECK 2024-09-16T07:38:21.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:21.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:21.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj197 = obj250 ']' 2024-09-16T07:38:21.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj197 td/test-erasure-eio/CHECK 2024-09-16T07:38:21.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:21.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:21.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj198 = obj250 ']' 2024-09-16T07:38:21.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj198 td/test-erasure-eio/CHECK 2024-09-16T07:38:21.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:21.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:21.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj199 = obj250 ']' 2024-09-16T07:38:21.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj199 td/test-erasure-eio/CHECK 2024-09-16T07:38:21.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:21.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:21.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj200 = obj250 ']' 2024-09-16T07:38:21.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj200 td/test-erasure-eio/CHECK 2024-09-16T07:38:21.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:21.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:21.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj201 = obj250 ']' 2024-09-16T07:38:21.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj201 td/test-erasure-eio/CHECK 2024-09-16T07:38:21.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:21.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:21.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj202 = obj250 ']' 2024-09-16T07:38:21.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj202 td/test-erasure-eio/CHECK 2024-09-16T07:38:21.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:21.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:21.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj203 = obj250 ']' 2024-09-16T07:38:21.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj203 td/test-erasure-eio/CHECK 2024-09-16T07:38:21.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:21.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:21.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj204 = obj250 ']' 2024-09-16T07:38:21.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj204 td/test-erasure-eio/CHECK 2024-09-16T07:38:21.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:21.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:21.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj205 = obj250 ']' 2024-09-16T07:38:21.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj205 td/test-erasure-eio/CHECK 2024-09-16T07:38:21.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:21.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:21.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj206 = obj250 ']' 2024-09-16T07:38:21.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj206 td/test-erasure-eio/CHECK 2024-09-16T07:38:21.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:21.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:21.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj207 = obj250 ']' 2024-09-16T07:38:21.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj207 td/test-erasure-eio/CHECK 2024-09-16T07:38:21.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:21.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:21.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj208 = obj250 ']' 2024-09-16T07:38:21.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj208 td/test-erasure-eio/CHECK 2024-09-16T07:38:21.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:21.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:21.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj209 = obj250 ']' 2024-09-16T07:38:21.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj209 td/test-erasure-eio/CHECK 2024-09-16T07:38:21.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:21.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:21.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj210 = obj250 ']' 2024-09-16T07:38:21.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj210 td/test-erasure-eio/CHECK 2024-09-16T07:38:21.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:21.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:21.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj211 = obj250 ']' 2024-09-16T07:38:21.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj211 td/test-erasure-eio/CHECK 2024-09-16T07:38:21.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:21.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:21.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj212 = obj250 ']' 2024-09-16T07:38:21.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj212 td/test-erasure-eio/CHECK 2024-09-16T07:38:22.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:22.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:22.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj213 = obj250 ']' 2024-09-16T07:38:22.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj213 td/test-erasure-eio/CHECK 2024-09-16T07:38:22.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:22.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:22.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj214 = obj250 ']' 2024-09-16T07:38:22.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj214 td/test-erasure-eio/CHECK 2024-09-16T07:38:22.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:22.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:22.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj215 = obj250 ']' 2024-09-16T07:38:22.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj215 td/test-erasure-eio/CHECK 2024-09-16T07:38:22.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:22.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:22.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj216 = obj250 ']' 2024-09-16T07:38:22.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj216 td/test-erasure-eio/CHECK 2024-09-16T07:38:22.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:22.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:22.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj217 = obj250 ']' 2024-09-16T07:38:22.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj217 td/test-erasure-eio/CHECK 2024-09-16T07:38:22.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:22.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:22.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj218 = obj250 ']' 2024-09-16T07:38:22.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj218 td/test-erasure-eio/CHECK 2024-09-16T07:38:22.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:22.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:22.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj219 = obj250 ']' 2024-09-16T07:38:22.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj219 td/test-erasure-eio/CHECK 2024-09-16T07:38:22.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:22.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:22.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj220 = obj250 ']' 2024-09-16T07:38:22.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj220 td/test-erasure-eio/CHECK 2024-09-16T07:38:22.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:22.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:22.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj221 = obj250 ']' 2024-09-16T07:38:22.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj221 td/test-erasure-eio/CHECK 2024-09-16T07:38:22.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:22.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:22.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj222 = obj250 ']' 2024-09-16T07:38:22.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj222 td/test-erasure-eio/CHECK 2024-09-16T07:38:22.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:22.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:22.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj223 = obj250 ']' 2024-09-16T07:38:22.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj223 td/test-erasure-eio/CHECK 2024-09-16T07:38:22.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:22.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:22.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj224 = obj250 ']' 2024-09-16T07:38:22.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj224 td/test-erasure-eio/CHECK 2024-09-16T07:38:22.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:22.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:22.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj225 = obj250 ']' 2024-09-16T07:38:22.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj225 td/test-erasure-eio/CHECK 2024-09-16T07:38:22.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:22.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:22.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj226 = obj250 ']' 2024-09-16T07:38:22.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj226 td/test-erasure-eio/CHECK 2024-09-16T07:38:22.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:22.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:22.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj227 = obj250 ']' 2024-09-16T07:38:22.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj227 td/test-erasure-eio/CHECK 2024-09-16T07:38:22.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:22.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:22.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj228 = obj250 ']' 2024-09-16T07:38:22.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj228 td/test-erasure-eio/CHECK 2024-09-16T07:38:22.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:22.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:22.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj229 = obj250 ']' 2024-09-16T07:38:22.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj229 td/test-erasure-eio/CHECK 2024-09-16T07:38:22.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:22.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:22.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj230 = obj250 ']' 2024-09-16T07:38:22.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj230 td/test-erasure-eio/CHECK 2024-09-16T07:38:22.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:22.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:22.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj231 = obj250 ']' 2024-09-16T07:38:22.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj231 td/test-erasure-eio/CHECK 2024-09-16T07:38:23.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:23.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:23.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj232 = obj250 ']' 2024-09-16T07:38:23.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj232 td/test-erasure-eio/CHECK 2024-09-16T07:38:23.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:23.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:23.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj233 = obj250 ']' 2024-09-16T07:38:23.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj233 td/test-erasure-eio/CHECK 2024-09-16T07:38:23.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:23.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:23.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj234 = obj250 ']' 2024-09-16T07:38:23.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj234 td/test-erasure-eio/CHECK 2024-09-16T07:38:23.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:23.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:23.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj235 = obj250 ']' 2024-09-16T07:38:23.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj235 td/test-erasure-eio/CHECK 2024-09-16T07:38:23.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:23.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:23.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj236 = obj250 ']' 2024-09-16T07:38:23.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj236 td/test-erasure-eio/CHECK 2024-09-16T07:38:23.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:23.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:23.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj237 = obj250 ']' 2024-09-16T07:38:23.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj237 td/test-erasure-eio/CHECK 2024-09-16T07:38:23.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:23.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:23.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj238 = obj250 ']' 2024-09-16T07:38:23.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj238 td/test-erasure-eio/CHECK 2024-09-16T07:38:23.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:23.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:23.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj239 = obj250 ']' 2024-09-16T07:38:23.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj239 td/test-erasure-eio/CHECK 2024-09-16T07:38:23.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:23.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:23.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj240 = obj250 ']' 2024-09-16T07:38:23.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj240 td/test-erasure-eio/CHECK 2024-09-16T07:38:23.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:23.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:23.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj241 = obj250 ']' 2024-09-16T07:38:23.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj241 td/test-erasure-eio/CHECK 2024-09-16T07:38:23.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:23.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:23.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj242 = obj250 ']' 2024-09-16T07:38:23.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj242 td/test-erasure-eio/CHECK 2024-09-16T07:38:23.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:23.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:23.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj243 = obj250 ']' 2024-09-16T07:38:23.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj243 td/test-erasure-eio/CHECK 2024-09-16T07:38:23.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:23.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:23.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj244 = obj250 ']' 2024-09-16T07:38:23.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj244 td/test-erasure-eio/CHECK 2024-09-16T07:38:23.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:23.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:23.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj245 = obj250 ']' 2024-09-16T07:38:23.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj245 td/test-erasure-eio/CHECK 2024-09-16T07:38:23.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:23.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:23.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj246 = obj250 ']' 2024-09-16T07:38:23.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj246 td/test-erasure-eio/CHECK 2024-09-16T07:38:23.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:23.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:23.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj247 = obj250 ']' 2024-09-16T07:38:23.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj247 td/test-erasure-eio/CHECK 2024-09-16T07:38:23.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:23.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:23.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj248 = obj250 ']' 2024-09-16T07:38:23.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj248 td/test-erasure-eio/CHECK 2024-09-16T07:38:23.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:23.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:23.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj249 = obj250 ']' 2024-09-16T07:38:23.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj249 td/test-erasure-eio/CHECK 2024-09-16T07:38:23.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:23.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:23.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj250 = obj250 ']' 2024-09-16T07:38:23.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2024-09-16T07:38:23.930 INFO:tasks.workunit.client.0.smithi162.stderr:error getting pool-jerasure/obj250: (2) No such file or directory 2024-09-16T07:38:23.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:23.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj251 = obj250 ']' 2024-09-16T07:38:23.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj251 td/test-erasure-eio/CHECK 2024-09-16T07:38:23.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:23.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:23.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj252 = obj250 ']' 2024-09-16T07:38:23.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj252 td/test-erasure-eio/CHECK 2024-09-16T07:38:24.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:24.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:24.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj253 = obj250 ']' 2024-09-16T07:38:24.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj253 td/test-erasure-eio/CHECK 2024-09-16T07:38:24.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:24.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:24.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj254 = obj250 ']' 2024-09-16T07:38:24.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj254 td/test-erasure-eio/CHECK 2024-09-16T07:38:24.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:24.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:24.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj255 = obj250 ']' 2024-09-16T07:38:24.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj255 td/test-erasure-eio/CHECK 2024-09-16T07:38:24.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:24.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:24.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj256 = obj250 ']' 2024-09-16T07:38:24.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj256 td/test-erasure-eio/CHECK 2024-09-16T07:38:24.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:24.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:24.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj257 = obj250 ']' 2024-09-16T07:38:24.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj257 td/test-erasure-eio/CHECK 2024-09-16T07:38:24.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:24.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:24.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj258 = obj250 ']' 2024-09-16T07:38:24.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj258 td/test-erasure-eio/CHECK 2024-09-16T07:38:24.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:24.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:24.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj259 = obj250 ']' 2024-09-16T07:38:24.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj259 td/test-erasure-eio/CHECK 2024-09-16T07:38:24.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:24.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:24.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj260 = obj250 ']' 2024-09-16T07:38:24.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj260 td/test-erasure-eio/CHECK 2024-09-16T07:38:24.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:24.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:24.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj261 = obj250 ']' 2024-09-16T07:38:24.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj261 td/test-erasure-eio/CHECK 2024-09-16T07:38:24.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:24.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:24.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj262 = obj250 ']' 2024-09-16T07:38:24.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj262 td/test-erasure-eio/CHECK 2024-09-16T07:38:24.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:24.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:24.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj263 = obj250 ']' 2024-09-16T07:38:24.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj263 td/test-erasure-eio/CHECK 2024-09-16T07:38:24.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:24.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:24.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj264 = obj250 ']' 2024-09-16T07:38:24.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj264 td/test-erasure-eio/CHECK 2024-09-16T07:38:24.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:24.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:24.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj265 = obj250 ']' 2024-09-16T07:38:24.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj265 td/test-erasure-eio/CHECK 2024-09-16T07:38:24.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:24.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:24.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj266 = obj250 ']' 2024-09-16T07:38:24.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj266 td/test-erasure-eio/CHECK 2024-09-16T07:38:24.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:24.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:24.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj267 = obj250 ']' 2024-09-16T07:38:24.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj267 td/test-erasure-eio/CHECK 2024-09-16T07:38:24.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:24.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:24.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj268 = obj250 ']' 2024-09-16T07:38:24.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj268 td/test-erasure-eio/CHECK 2024-09-16T07:38:24.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:24.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:24.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj269 = obj250 ']' 2024-09-16T07:38:24.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj269 td/test-erasure-eio/CHECK 2024-09-16T07:38:24.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:24.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:24.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj270 = obj250 ']' 2024-09-16T07:38:24.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj270 td/test-erasure-eio/CHECK 2024-09-16T07:38:24.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:24.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:24.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj271 = obj250 ']' 2024-09-16T07:38:24.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj271 td/test-erasure-eio/CHECK 2024-09-16T07:38:25.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:25.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:25.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj272 = obj250 ']' 2024-09-16T07:38:25.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj272 td/test-erasure-eio/CHECK 2024-09-16T07:38:25.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:25.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:25.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj273 = obj250 ']' 2024-09-16T07:38:25.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj273 td/test-erasure-eio/CHECK 2024-09-16T07:38:25.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:25.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:25.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj274 = obj250 ']' 2024-09-16T07:38:25.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj274 td/test-erasure-eio/CHECK 2024-09-16T07:38:25.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:25.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:25.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj275 = obj250 ']' 2024-09-16T07:38:25.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj275 td/test-erasure-eio/CHECK 2024-09-16T07:38:25.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:25.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:25.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj276 = obj250 ']' 2024-09-16T07:38:25.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj276 td/test-erasure-eio/CHECK 2024-09-16T07:38:25.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:25.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:25.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj277 = obj250 ']' 2024-09-16T07:38:25.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj277 td/test-erasure-eio/CHECK 2024-09-16T07:38:25.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:25.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:25.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj278 = obj250 ']' 2024-09-16T07:38:25.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj278 td/test-erasure-eio/CHECK 2024-09-16T07:38:25.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:25.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:25.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj279 = obj250 ']' 2024-09-16T07:38:25.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj279 td/test-erasure-eio/CHECK 2024-09-16T07:38:25.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:25.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:25.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj280 = obj250 ']' 2024-09-16T07:38:25.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj280 td/test-erasure-eio/CHECK 2024-09-16T07:38:25.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:25.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:25.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj281 = obj250 ']' 2024-09-16T07:38:25.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj281 td/test-erasure-eio/CHECK 2024-09-16T07:38:25.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:25.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:25.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj282 = obj250 ']' 2024-09-16T07:38:25.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj282 td/test-erasure-eio/CHECK 2024-09-16T07:38:25.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:25.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:25.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj283 = obj250 ']' 2024-09-16T07:38:25.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj283 td/test-erasure-eio/CHECK 2024-09-16T07:38:25.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:25.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:25.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj284 = obj250 ']' 2024-09-16T07:38:25.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj284 td/test-erasure-eio/CHECK 2024-09-16T07:38:25.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:25.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:25.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj285 = obj250 ']' 2024-09-16T07:38:25.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj285 td/test-erasure-eio/CHECK 2024-09-16T07:38:25.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:25.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:25.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj286 = obj250 ']' 2024-09-16T07:38:25.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj286 td/test-erasure-eio/CHECK 2024-09-16T07:38:25.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:25.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:25.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj287 = obj250 ']' 2024-09-16T07:38:25.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj287 td/test-erasure-eio/CHECK 2024-09-16T07:38:25.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:25.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:25.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj288 = obj250 ']' 2024-09-16T07:38:25.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj288 td/test-erasure-eio/CHECK 2024-09-16T07:38:25.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:25.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:25.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj289 = obj250 ']' 2024-09-16T07:38:25.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj289 td/test-erasure-eio/CHECK 2024-09-16T07:38:25.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:25.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:25.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj290 = obj250 ']' 2024-09-16T07:38:25.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj290 td/test-erasure-eio/CHECK 2024-09-16T07:38:25.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:25.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:25.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj291 = obj250 ']' 2024-09-16T07:38:25.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj291 td/test-erasure-eio/CHECK 2024-09-16T07:38:25.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:25.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:25.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj292 = obj250 ']' 2024-09-16T07:38:25.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj292 td/test-erasure-eio/CHECK 2024-09-16T07:38:26.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:26.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:26.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj293 = obj250 ']' 2024-09-16T07:38:26.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj293 td/test-erasure-eio/CHECK 2024-09-16T07:38:26.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:26.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:26.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj294 = obj250 ']' 2024-09-16T07:38:26.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj294 td/test-erasure-eio/CHECK 2024-09-16T07:38:26.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:26.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:26.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj295 = obj250 ']' 2024-09-16T07:38:26.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj295 td/test-erasure-eio/CHECK 2024-09-16T07:38:26.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:26.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:26.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj296 = obj250 ']' 2024-09-16T07:38:26.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj296 td/test-erasure-eio/CHECK 2024-09-16T07:38:26.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:26.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:26.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj297 = obj250 ']' 2024-09-16T07:38:26.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj297 td/test-erasure-eio/CHECK 2024-09-16T07:38:26.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:26.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:26.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj298 = obj250 ']' 2024-09-16T07:38:26.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj298 td/test-erasure-eio/CHECK 2024-09-16T07:38:26.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:26.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:26.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj299 = obj250 ']' 2024-09-16T07:38:26.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj299 td/test-erasure-eio/CHECK 2024-09-16T07:38:26.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:26.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-16T07:38:26.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj300 = obj250 ']' 2024-09-16T07:38:26.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj300 td/test-erasure-eio/CHECK 2024-09-16T07:38:26.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:26.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:602: TEST_ec_backfill_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T07:38:26.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:604: TEST_ec_backfill_unfound: delete_erasure_coded_pool pool-jerasure 2024-09-16T07:38:26.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T07:38:26.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-16T07:38:27.081 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-16T07:38:27.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-16T07:38:27.944 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-16T07:38:27.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-16T07:38:27.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T07:38:27.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:38:27.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T07:38:27.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:38:27.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:38:27.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:38:27.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:38:27.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:38:28.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:38:28.274 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:38:28.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:38:28.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:38:28.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:38:28.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:38:28.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:38:28.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:38:28.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:38:28.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:38:28.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:38:28.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:38:28.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:38:28.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T07:38:28.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:38:28.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:38:28.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:38:28.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T07:38:28.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:38:28.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:38:28.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-16T07:38:28.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-16T07:38:28.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-16T07:38:28.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-16T07:38:28.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T07:38:28.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:38:28.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T07:38:28.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:38:28.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:38:28.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:38:28.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:38:28.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:38:28.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:38:28.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:38:28.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:38:28.352 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:38:28.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:38:28.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:38:28.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:38:28.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:38:28.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:38:28.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:38:28.358 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:38:28.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:38:28.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:38:28.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T07:38:28.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:38:28.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:38:28.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:38:28.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T07:38:28.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:38:28.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:38:28.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-16T07:38:28.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T07:38:28.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:38:28.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:38:28.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79228 2024-09-16T07:38:28.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T07:38:28.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T07:38:28.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T07:38:28.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-16T07:38:28.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-16T07:38:28.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-16T07:38:28.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T07:38:28.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T07:38:28.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T07:38:28.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-16T07:38:28.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-16T07:38:28.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T07:38:28.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:38:28.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:38:28.828 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:38:28.829 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:38:28.829 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:38:28.829 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:38:28.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T07:38:29.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T07:38:29.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T07:38:29.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:38:29.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:38:29.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T07:38:29.057 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T07:38:29.058 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:38:29.058 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:38:29.058 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:38:29.059 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:38:29.059 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:38:29.059 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:38:29.059 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T07:38:29.059 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:38:29.060 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get fsid 2024-09-16T07:38:29.161 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T07:38:29.161 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:38:29.161 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:38:29.161 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T07:38:29.162 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T07:38:29.163 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:38:29.163 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:38:29.163 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:38:29.163 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:38:29.164 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:38:29.164 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:38:29.164 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T07:38:29.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:38:29.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get mon_host 2024-09-16T07:38:29.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-16T07:38:29.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-16T07:38:29.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T07:38:29.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T07:38:29.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T07:38:29.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-16T07:38:29.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T07:38:30.163 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T07:38:30.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:38:30.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:38:30.164 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:38:30.164 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:38:30.164 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:38:30.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:38:30.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:38:30.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:38:30.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-16T07:38:30.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T07:38:30.512 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-16T07:38:30.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T07:38:31.524 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-16T07:38:31.524 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:38:31.524 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:38:31.525 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:38:31.525 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:38:31.525 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:38:31.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T07:38:31.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-16T07:38:31.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-mon.a.asok log flush 2024-09-16T07:38:31.609 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:38:31.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-16T07:38:31.620 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:38:28.881+0000 7f8987d34d40 0 load: jerasure load: lrc 2024-09-16T07:38:31.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_object_attr_read_error td/test-erasure-eio 2024-09-16T07:38:31.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:359: TEST_ec_object_attr_read_error: local dir=td/test-erasure-eio 2024-09-16T07:38:31.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:360: TEST_ec_object_attr_read_error: local objname=myobject 2024-09-16T07:38:31.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:362: TEST_ec_object_attr_read_error: setup_osds 7 2024-09-16T07:38:31.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-09-16T07:38:31.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-16T07:38:31.622 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-09-16T07:38:31.623 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-09-16T07:38:31.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:38:31.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-16T07:38:31.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:38:31.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:38:31.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T07:38:31.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:38:31.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-16T07:38:31.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:38:31.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:38:31.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:38:31.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:38:31.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-16T07:38:31.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-16T07:38:31.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:38:31.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:38:31.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:38:31.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:38:31.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:38:31.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:38:31.628 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:38:31.628 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:38:31.628 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:38:31.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:38:31.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:38:31.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:38:31.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:38:31.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:38:31.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:38:31.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:38:31.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:38:31.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:38:31.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:38:31.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:38:31.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:38:31.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:38:31.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-16T07:38:31.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:38:31.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2072ee76-cdc0-4afb-b1ba-6e958fda349f 2024-09-16T07:38:31.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2072ee76-cdc0-4afb-b1ba-6e958fda349f' 2024-09-16T07:38:31.632 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 2072ee76-cdc0-4afb-b1ba-6e958fda349f 2024-09-16T07:38:31.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:38:31.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB34OdmDRb4JhAAHET2ImvKAVNGX8XNobNpGQ== 2024-09-16T07:38:31.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB34OdmDRb4JhAAHET2ImvKAVNGX8XNobNpGQ=="}' 2024-09-16T07:38:31.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2072ee76-cdc0-4afb-b1ba-6e958fda349f -i td/test-erasure-eio/0/new.json 2024-09-16T07:38:32.009 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:38:32.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-16T07:38:32.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB34OdmDRb4JhAAHET2ImvKAVNGX8XNobNpGQ== --osd-uuid 2072ee76-cdc0-4afb-b1ba-6e958fda349f 2024-09-16T07:38:32.065 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:38:32.061+0000 7fa485864880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:38:32.068 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:38:32.065+0000 7fa485864880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:38:32.070 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:38:32.069+0000 7fa485864880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:38:32.071 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:38:32.069+0000 7fa485864880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T07:38:32.072 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:38:32.069+0000 7fa485864880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T07:38:32.072 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:38:32.069+0000 7fa485864880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T07:38:32.157 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:38:32.153+0000 7fa485864880 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-16T07:38:37.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-16T07:38:37.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:38:37.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T07:38:37.841 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T07:38:37.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:38:38.408 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T07:38:38.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T07:38:38.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:38:38.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:38:38.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:38:38.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:38:38.444 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:38:38.441+0000 7f6de5b4b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:38:38.464 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:38:38.461+0000 7f6de5b4b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:38:38.465 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:38:38.461+0000 7f6de5b4b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:38:38.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T07:38:38.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:38:38.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T07:38:38.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:38:38.845 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:38:38.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:38:38.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:38:38.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:38:38.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:38:38.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:38:39.037 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:38:39.033+0000 7f6de5b4b880 -1 Falling back to public interface 2024-09-16T07:38:39.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:38:40.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:38:40.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:38:40.286 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:38:40.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:38:40.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:38:40.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:38:40.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:38:41.639 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:38:41.637+0000 7f6de5b4b880 -1 osd.0 0 log_to_monitors true 2024-09-16T07:38:41.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:38:41.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:38:41.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:38:41.692 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:38:41.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:38:41.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:38:42.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:38:43.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:38:43.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:38:43.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:38:43.218 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:38:43.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:38:43.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:38:43.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:38:44.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:38:44.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:38:44.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:38:44.656 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:38:44.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:38:44.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:38:45.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:38:46.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:38:46.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:38:46.146 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:38:46.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:38:46.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:38:46.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:38:46.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:38:47.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:38:47.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:38:47.636 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:38:47.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:38:47.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:38:47.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:38:48.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:38:49.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:38:49.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:38:49.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:38:49.112 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:38:49.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:38:49.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:38:49.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:38:50.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:38:50.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:38:50.649 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:38:50.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:38:50.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:38:50.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:38:51.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:38:52.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:38:52.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:38:52.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:38:52.082 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:38:52.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:38:52.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:38:52.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:38:53.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:38:53.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:38:53.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:38:53.659 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:38:53.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:38:53.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:38:54.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:38:55.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:38:55.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:38:55.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:38:55.173 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:38:55.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:38:55.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:38:56.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:38:57.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:38:57.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:38:57.162 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:38:57.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:38:57.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:38:57.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:38:57.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:38:58.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:38:58.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:38:58.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:38:58.975 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:38:58.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:38:58.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:38:59.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:00.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:00.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:00.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:39:00.553 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:39:00.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:00.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:39:01.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:02.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:02.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:02.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:39:02.133 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:39:02.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:02.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:39:02.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:03.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:03.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:03.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:39:03.913 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:39:03.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:39:03.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:04.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:05.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:05.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:05.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:39:05.873 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:39:05.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:05.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:39:06.381 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3196980735,v1:127.0.0.1:6803/3196980735] [v2:127.0.0.1:6804/3196980735,v1:127.0.0.1:6805/3196980735] exists,up 2072ee76-cdc0-4afb-b1ba-6e958fda349f 2024-09-16T07:39:06.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:39:06.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:39:06.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:39:06.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:39:06.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-16T07:39:06.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:39:06.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:39:06.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T07:39:06.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:39:06.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-16T07:39:06.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:39:06.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:39:06.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:39:06.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:39:06.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-16T07:39:06.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-16T07:39:06.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:39:06.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:39:06.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:39:06.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:39:06.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:39:06.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:39:06.385 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:39:06.385 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:39:06.385 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:39:06.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:39:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:39:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:39:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:39:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:39:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:39:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:39:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:39:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:39:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:39:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:39:06.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:39:06.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:39:06.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-16T07:39:06.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:39:06.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=83fd47d7-186c-4e12-9cf1-55ebd9b37910 2024-09-16T07:39:06.389 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 83fd47d7-186c-4e12-9cf1-55ebd9b37910 2024-09-16T07:39:06.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 83fd47d7-186c-4e12-9cf1-55ebd9b37910' 2024-09-16T07:39:06.389 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:39:06.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCa4OdmfiKlGBAAfMAFHv+XxXoQWCpUZuFJEg== 2024-09-16T07:39:06.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCa4OdmfiKlGBAAfMAFHv+XxXoQWCpUZuFJEg=="}' 2024-09-16T07:39:06.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 83fd47d7-186c-4e12-9cf1-55ebd9b37910 -i td/test-erasure-eio/1/new.json 2024-09-16T07:39:06.946 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:39:06.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-16T07:39:06.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCa4OdmfiKlGBAAfMAFHv+XxXoQWCpUZuFJEg== --osd-uuid 83fd47d7-186c-4e12-9cf1-55ebd9b37910 2024-09-16T07:39:07.001 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:39:06.996+0000 7fbfcf745880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:39:07.035 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:39:07.032+0000 7fbfcf745880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:39:07.037 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:39:07.032+0000 7fbfcf745880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:39:07.038 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:39:07.036+0000 7fbfcf745880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T07:39:07.038 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:39:07.036+0000 7fbfcf745880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T07:39:07.039 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:39:07.036+0000 7fbfcf745880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T07:39:07.160 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:39:07.156+0000 7fbfcf745880 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-16T07:39:13.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-16T07:39:13.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:39:13.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T07:39:13.169 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-16T07:39:13.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:39:13.734 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-16T07:39:13.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T07:39:13.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:39:13.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:39:13.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:39:13.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:39:13.769 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:39:13.764+0000 7f9af7817880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:39:13.857 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:39:13.852+0000 7f9af7817880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:39:13.859 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:39:13.856+0000 7f9af7817880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:39:14.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T07:39:14.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:39:14.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T07:39:14.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:39:14.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:39:14.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:14.171 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:39:14.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:39:14.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:14.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:39:14.421 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:39:14.416+0000 7f9af7817880 -1 Falling back to public interface 2024-09-16T07:39:14.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:15.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:15.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:15.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:39:15.668 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:39:15.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:15.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:39:16.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:16.976 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:39:16.972+0000 7f9af7817880 -1 osd.1 0 log_to_monitors true 2024-09-16T07:39:17.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:17.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:17.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:39:17.096 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:39:17.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:17.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:39:17.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:18.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:18.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:18.672 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:39:18.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:39:18.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:39:18.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:19.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:20.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:20.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:20.170 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:39:20.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:39:20.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:20.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:39:20.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:21.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:21.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:21.700 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:39:21.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:39:21.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:21.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:39:22.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:23.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:23.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:23.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:39:23.218 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:39:23.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:23.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:39:23.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:24.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:24.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:24.761 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:39:24.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:39:24.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:24.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:39:25.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:26.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:26.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:26.204 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:39:26.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:39:26.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:39:26.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:26.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:27.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:27.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:27.683 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:39:27.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:39:27.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:27.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:39:28.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:29.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:29.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:29.179 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:39:29.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:39:29.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:29.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:39:29.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:30.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:30.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:30.675 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:39:30.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:39:30.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:30.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:39:31.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:32.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:32.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:32.472 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:39:32.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:39:32.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:32.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:39:33.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:34.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:34.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:34.232 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:39:34.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:39:34.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:34.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:39:35.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:36.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:36.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:36.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:39:36.053 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:39:36.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:36.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:39:36.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:37.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:37.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:37.683 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:39:37.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:39:37.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:37.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:39:38.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:39.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:39.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:39.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:39:39.697 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:39:39.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:39.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:39:40.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:41.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:41.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:41.469 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:39:41.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:39:41.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:41.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:39:42.090 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4195720005,v1:127.0.0.1:6811/4195720005] [v2:127.0.0.1:6812/4195720005,v1:127.0.0.1:6813/4195720005] exists,up 83fd47d7-186c-4e12-9cf1-55ebd9b37910 2024-09-16T07:39:42.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:39:42.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:39:42.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:39:42.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:39:42.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-16T07:39:42.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:39:42.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:39:42.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T07:39:42.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:39:42.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-16T07:39:42.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:39:42.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:39:42.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:39:42.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:39:42.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-16T07:39:42.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-16T07:39:42.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:39:42.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:39:42.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:39:42.094 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:39:42.094 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:39:42.094 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:39:42.094 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:39:42.094 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:39:42.094 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:39:42.094 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:39:42.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:39:42.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:39:42.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:39:42.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:39:42.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:39:42.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:39:42.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:39:42.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:39:42.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:39:42.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:39:42.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:39:42.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:39:42.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-16T07:39:42.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:39:42.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=33ea9716-3f5e-42ab-bf73-e0904df6e04b 2024-09-16T07:39:42.098 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 33ea9716-3f5e-42ab-bf73-e0904df6e04b 2024-09-16T07:39:42.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 33ea9716-3f5e-42ab-bf73-e0904df6e04b' 2024-09-16T07:39:42.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:39:42.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC+4OdmBBE0BxAAkv/d/xPxam/icduB+YJjkQ== 2024-09-16T07:39:42.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC+4OdmBBE0BxAAkv/d/xPxam/icduB+YJjkQ=="}' 2024-09-16T07:39:42.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 33ea9716-3f5e-42ab-bf73-e0904df6e04b -i td/test-erasure-eio/2/new.json 2024-09-16T07:39:42.951 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:39:42.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-16T07:39:42.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC+4OdmBBE0BxAAkv/d/xPxam/icduB+YJjkQ== --osd-uuid 33ea9716-3f5e-42ab-bf73-e0904df6e04b 2024-09-16T07:39:43.004 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:39:42.999+0000 7f3740857880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:39:43.083 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:39:43.079+0000 7f3740857880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:39:43.085 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:39:43.079+0000 7f3740857880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:39:43.086 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:39:43.083+0000 7f3740857880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T07:39:43.086 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:39:43.083+0000 7f3740857880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T07:39:43.086 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:39:43.083+0000 7f3740857880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T07:39:43.209 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:39:43.207+0000 7f3740857880 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-16T07:39:49.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-16T07:39:49.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:39:49.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T07:39:49.677 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-16T07:39:49.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:39:50.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T07:39:50.276 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-16T07:39:50.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:39:50.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:39:50.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:39:50.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:39:50.309 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:39:50.303+0000 7f98c622a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:39:50.325 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:39:50.319+0000 7f98c622a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:39:50.327 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:39:50.323+0000 7f98c622a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:39:50.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T07:39:50.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:39:50.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T07:39:50.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:39:50.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:39:50.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:50.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:39:50.724 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:39:50.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:50.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:39:50.905 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:39:50.899+0000 7f98c622a880 -1 Falling back to public interface 2024-09-16T07:39:51.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:52.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:52.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:52.147 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:39:52.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:39:52.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:52.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:39:52.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:53.446 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:39:53.439+0000 7f98c622a880 -1 osd.2 0 log_to_monitors true 2024-09-16T07:39:53.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:53.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:53.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:39:53.576 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:39:53.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:53.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:39:54.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:55.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:55.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:55.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:39:55.090 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:39:55.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:55.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:39:55.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:56.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:56.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:56.589 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:39:56.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:39:56.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:56.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:39:57.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:58.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:58.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:58.331 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:39:58.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:39:58.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:58.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:39:58.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:39:59.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:39:59.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:39:59.755 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:39:59.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:39:59.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:39:59.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:40:00.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:01.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:01.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:01.241 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:40:01.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:40:01.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:01.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:40:01.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:02.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:02.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:02.672 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:40:02.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:40:02.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:02.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:40:03.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:04.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:04.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:04.254 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:40:04.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:40:04.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:04.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:40:04.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:05.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:05.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:05.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:40:05.771 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:40:05.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:05.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:40:06.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:07.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:07.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:07.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:40:07.247 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:40:07.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:07.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:40:07.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:08.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:08.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:08.853 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:40:08.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:40:08.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:08.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:40:10.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:11.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:11.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:11.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:40:11.054 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:40:11.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:11.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:40:11.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:12.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:12.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:12.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:40:12.657 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:40:12.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:40:12.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:13.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:14.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:14.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:14.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:40:14.414 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:40:14.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:14.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:40:15.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:16.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:16.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:16.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:40:16.418 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:40:16.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:16.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:40:17.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:18.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:18.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:18.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:40:18.080 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:40:18.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:18.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:40:18.617 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2794937275,v1:127.0.0.1:6819/2794937275] [v2:127.0.0.1:6820/2794937275,v1:127.0.0.1:6821/2794937275] exists,up 33ea9716-3f5e-42ab-bf73-e0904df6e04b 2024-09-16T07:40:18.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:40:18.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:40:18.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:40:18.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:40:18.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-16T07:40:18.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:40:18.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:40:18.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T07:40:18.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:40:18.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-16T07:40:18.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:40:18.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:40:18.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:40:18.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:40:18.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-16T07:40:18.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-16T07:40:18.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:40:18.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:40:18.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:40:18.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:40:18.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:40:18.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:40:18.622 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:40:18.622 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:40:18.622 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:40:18.623 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:40:18.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:40:18.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:40:18.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:40:18.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:40:18.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:40:18.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:40:18.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:40:18.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:40:18.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:40:18.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:40:18.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:40:18.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:40:18.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-16T07:40:18.625 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:40:18.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5619885f-c5d9-495f-953f-567d45139df6 2024-09-16T07:40:18.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 5619885f-c5d9-495f-953f-567d45139df6' 2024-09-16T07:40:18.627 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 5619885f-c5d9-495f-953f-567d45139df6 2024-09-16T07:40:18.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:40:18.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDi4Odm9rn3JhAAATVZUa9FGuR4h+UsI9YqiA== 2024-09-16T07:40:18.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDi4Odm9rn3JhAAATVZUa9FGuR4h+UsI9YqiA=="}' 2024-09-16T07:40:18.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5619885f-c5d9-495f-953f-567d45139df6 -i td/test-erasure-eio/3/new.json 2024-09-16T07:40:19.684 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:40:19.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-16T07:40:19.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDi4Odm9rn3JhAAATVZUa9FGuR4h+UsI9YqiA== --osd-uuid 5619885f-c5d9-495f-953f-567d45139df6 2024-09-16T07:40:19.743 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:40:19.739+0000 7f3c4727b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:40:19.747 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:40:19.743+0000 7f3c4727b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:40:19.749 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:40:19.743+0000 7f3c4727b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:40:19.750 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:40:19.743+0000 7f3c4727b880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T07:40:19.750 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:40:19.743+0000 7f3c4727b880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T07:40:19.750 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:40:19.743+0000 7f3c4727b880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T07:40:19.856 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:40:19.851+0000 7f3c4727b880 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-16T07:40:26.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-16T07:40:26.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:40:26.045 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-16T07:40:26.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T07:40:26.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:40:26.599 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-16T07:40:26.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T07:40:26.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:40:26.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:40:26.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:40:26.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:40:26.633 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:40:26.626+0000 7f3af7292880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:40:26.652 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:40:26.646+0000 7f3af7292880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:40:26.654 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:40:26.650+0000 7f3af7292880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:40:27.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T07:40:27.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:40:27.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T07:40:27.031 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:40:27.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:40:27.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:40:27.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:27.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:40:27.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:27.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:40:27.215 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:40:27.210+0000 7f3af7292880 -1 Falling back to public interface 2024-09-16T07:40:27.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:28.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:28.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:28.459 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:40:28.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:40:28.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:40:28.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:28.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:29.509 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:40:29.502+0000 7f3af7292880 -1 osd.3 0 log_to_monitors true 2024-09-16T07:40:29.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:29.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:29.863 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:40:29.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:40:29.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:29.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:40:30.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:31.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:31.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:31.537 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:40:31.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:40:31.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:31.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:40:32.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:33.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:33.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:33.211 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:40:33.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:40:33.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:33.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:40:33.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:34.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:34.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:34.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:40:34.717 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:40:34.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:34.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:40:35.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:36.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:36.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:36.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:40:36.336 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:40:36.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:40:36.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:36.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:37.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:37.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:37.821 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:40:37.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:40:37.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:37.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:40:38.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:39.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:39.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:39.398 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:40:39.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:40:39.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:39.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:40:39.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:40.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:40.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:40.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:40:40.857 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:40:40.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:40.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:40:41.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:42.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:42.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:42.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:40:42.447 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:40:42.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:42.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:40:43.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:44.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:44.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:44.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:40:44.064 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:40:44.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:44.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:40:45.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:46.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:46.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:46.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:40:46.176 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:40:46.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:46.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:40:46.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:47.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:47.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:47.878 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:40:47.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:40:47.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:47.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:40:48.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:49.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:49.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:49.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:40:49.537 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:40:49.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:49.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:40:50.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:51.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:51.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:51.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:40:51.252 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:40:51.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:51.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:40:52.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:53.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:53.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:53.437 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:40:53.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:40:53.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:53.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:40:54.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:40:55.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:40:55.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:40:55.190 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:40:55.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:40:55.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:40:55.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:40:55.784 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3931412768,v1:127.0.0.1:6827/3931412768] [v2:127.0.0.1:6828/3931412768,v1:127.0.0.1:6829/3931412768] exists,up 5619885f-c5d9-495f-953f-567d45139df6 2024-09-16T07:40:55.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:40:55.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:40:55.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:40:55.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:40:55.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-16T07:40:55.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:40:55.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:40:55.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-16T07:40:55.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:40:55.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-16T07:40:55.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:40:55.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:40:55.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:40:55.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:40:55.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-16T07:40:55.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-16T07:40:55.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:40:55.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:40:55.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:40:55.788 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:40:55.788 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:40:55.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:40:55.789 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:40:55.789 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:40:55.789 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:40:55.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:40:55.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:40:55.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:40:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:40:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:40:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:40:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:40:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:40:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:40:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:40:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:40:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:40:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:40:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-16T07:40:55.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:40:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=82254bba-3ca8-46db-952e-fbfc463d446a 2024-09-16T07:40:55.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 82254bba-3ca8-46db-952e-fbfc463d446a' 2024-09-16T07:40:55.792 INFO:tasks.workunit.client.0.smithi162.stdout:add osd4 82254bba-3ca8-46db-952e-fbfc463d446a 2024-09-16T07:40:55.792 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:40:55.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAH4edmgxyvMBAA+w9BpmXeSZApSSbtr8jcUg== 2024-09-16T07:40:55.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAH4edmgxyvMBAA+w9BpmXeSZApSSbtr8jcUg=="}' 2024-09-16T07:40:55.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 82254bba-3ca8-46db-952e-fbfc463d446a -i td/test-erasure-eio/4/new.json 2024-09-16T07:40:57.463 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:40:57.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-16T07:40:57.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAH4edmgxyvMBAA+w9BpmXeSZApSSbtr8jcUg== --osd-uuid 82254bba-3ca8-46db-952e-fbfc463d446a 2024-09-16T07:40:57.525 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:40:57.522+0000 7f2c6159b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:40:57.528 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:40:57.526+0000 7f2c6159b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:40:57.530 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:40:57.530+0000 7f2c6159b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:40:57.531 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:40:57.530+0000 7f2c6159b880 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-16T07:40:57.531 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:40:57.530+0000 7f2c6159b880 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-16T07:40:57.531 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:40:57.530+0000 7f2c6159b880 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-16T07:40:57.700 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:40:57.698+0000 7f2c6159b880 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-16T07:41:04.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-16T07:41:04.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:41:04.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-16T07:41:04.209 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd4 key to auth repository 2024-09-16T07:41:04.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:41:04.964 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.4 2024-09-16T07:41:04.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-16T07:41:04.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:41:04.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:41:04.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:41:04.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:41:04.999 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:41:04.998+0000 7f335d957880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:41:05.033 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:41:05.030+0000 7f335d957880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:41:05.036 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:41:05.034+0000 7f335d957880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:41:05.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-16T07:41:05.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:41:05.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-16T07:41:05.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:41:05.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:41:05.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:05.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:41:05.474 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:41:05.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:05.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:41:05.597 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:41:05.594+0000 7f335d957880 -1 Falling back to public interface 2024-09-16T07:41:06.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:07.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:07.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:07.023 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:41:07.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:41:07.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:07.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:41:07.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:08.274 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:41:08.274+0000 7f335d957880 -1 osd.4 0 log_to_monitors true 2024-09-16T07:41:08.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:08.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:08.497 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:41:08.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:41:08.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:08.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:41:09.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:10.168 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:41:10.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:10.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:10.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:41:10.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:10.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:41:10.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:11.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:11.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:11.880 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:41:11.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:41:11.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:11.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:41:12.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:13.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:13.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:13.439 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:41:13.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:41:13.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:13.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:41:13.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:14.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:14.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:14.876 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:41:14.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:41:14.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:14.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:41:15.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:16.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:16.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:16.474 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:41:16.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:41:16.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:16.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:41:17.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:18.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:18.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:18.169 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:41:18.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:41:18.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:18.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:41:18.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:19.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:19.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:19.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:41:19.707 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:41:19.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:19.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:41:20.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:21.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:21.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:21.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:41:21.234 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:41:21.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:21.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:41:21.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:22.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:22.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:22.681 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:41:22.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:41:22.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:22.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:41:23.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:24.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:24.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:24.190 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:41:24.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:41:24.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:24.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:41:24.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:25.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:25.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:25.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:41:25.682 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:41:25.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:41:25.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:26.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:27.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:27.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:27.695 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:41:27.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:41:27.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:27.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:41:28.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:29.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:29.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:29.320 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:41:29.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:41:29.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:29.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:41:29.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:30.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:30.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:30.914 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:41:30.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:41:30.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:30.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:41:31.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:32.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:32.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:32.835 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:41:32.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:41:32.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:32.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:41:33.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:34.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:34.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:34.335 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-16T07:41:34.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T07:41:34.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:34.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:41:34.772 INFO:tasks.workunit.client.0.smithi162.stdout:osd.4 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1319113882,v1:127.0.0.1:6835/1319113882] [v2:127.0.0.1:6836/1319113882,v1:127.0.0.1:6837/1319113882] exists,up 82254bba-3ca8-46db-952e-fbfc463d446a 2024-09-16T07:41:34.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:41:34.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:41:34.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:41:34.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:41:34.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-16T07:41:34.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:41:34.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:41:34.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-16T07:41:34.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:41:34.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-16T07:41:34.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:41:34.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:41:34.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:41:34.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:41:34.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-16T07:41:34.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-16T07:41:34.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:41:34.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:41:34.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:41:34.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:41:34.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:41:34.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:41:34.778 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:41:34.778 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:41:34.778 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:41:34.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:41:34.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:41:34.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:41:34.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:41:34.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:41:34.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:41:34.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:41:34.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:41:34.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:41:34.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:41:34.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:41:34.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:41:34.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:41:34.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-16T07:41:34.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:41:34.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a0b3f08c-376b-45a6-822d-9ae441faac35 2024-09-16T07:41:34.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 a0b3f08c-376b-45a6-822d-9ae441faac35' 2024-09-16T07:41:34.780 INFO:tasks.workunit.client.0.smithi162.stdout:add osd5 a0b3f08c-376b-45a6-822d-9ae441faac35 2024-09-16T07:41:34.780 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:41:34.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAu4edm2MuRLxAAF9ci5PIEKHtVbbhy5FJc1A== 2024-09-16T07:41:34.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAu4edm2MuRLxAAF9ci5PIEKHtVbbhy5FJc1A=="}' 2024-09-16T07:41:34.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a0b3f08c-376b-45a6-822d-9ae441faac35 -i td/test-erasure-eio/5/new.json 2024-09-16T07:41:35.938 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:41:35.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-16T07:41:35.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAu4edm2MuRLxAAF9ci5PIEKHtVbbhy5FJc1A== --osd-uuid a0b3f08c-376b-45a6-822d-9ae441faac35 2024-09-16T07:41:35.990 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:41:35.989+0000 7f6f3822d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:41:36.198 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:41:36.197+0000 7f6f3822d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:41:36.201 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:41:36.197+0000 7f6f3822d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:41:36.202 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:41:36.197+0000 7f6f3822d880 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-16T07:41:36.202 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:41:36.197+0000 7f6f3822d880 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-16T07:41:36.202 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:41:36.197+0000 7f6f3822d880 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-16T07:41:36.438 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:41:36.437+0000 7f6f3822d880 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-16T07:41:43.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-16T07:41:43.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:41:43.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-16T07:41:43.225 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd5 key to auth repository 2024-09-16T07:41:43.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:41:43.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-16T07:41:43.784 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.5 2024-09-16T07:41:43.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:41:43.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:41:43.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:41:43.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:41:43.820 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:41:43.817+0000 7ffaa6d55880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:41:43.845 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:41:43.841+0000 7ffaa6d55880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:41:43.846 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:41:43.845+0000 7ffaa6d55880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:41:44.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-16T07:41:44.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:41:44.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-16T07:41:44.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:41:44.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:41:44.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:44.211 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:41:44.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:41:44.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:44.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:41:44.415 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:41:44.413+0000 7ffaa6d55880 -1 Falling back to public interface 2024-09-16T07:41:44.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:45.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:45.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:45.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:41:45.660 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:41:45.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:45.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:41:46.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:46.870 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:41:46.869+0000 7ffaa6d55880 -1 osd.5 0 log_to_monitors true 2024-09-16T07:41:47.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:47.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:47.107 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:41:47.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:41:47.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:47.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:41:47.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:48.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:48.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:48.757 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:41:48.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:41:48.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:48.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:41:49.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:50.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:50.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:50.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:41:50.355 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:41:50.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:50.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:41:51.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:52.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:52.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:52.056 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:41:52.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:41:52.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:52.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:41:52.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:53.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:53.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:53.609 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:41:53.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:41:53.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:53.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:41:54.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:55.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:55.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:55.094 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:41:55.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:41:55.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:55.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:41:55.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:56.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:56.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:56.499 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:41:56.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:41:56.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:56.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:41:57.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:58.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:58.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:58.054 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:41:58.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:41:58.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:58.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:41:58.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:41:59.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:41:59.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:41:59.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:41:59.595 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:41:59.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:41:59.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:42:00.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:01.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:01.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:01.016 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:42:01.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:42:01.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:01.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:42:01.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:02.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:02.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:02.741 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:42:02.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:42:02.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:02.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:42:03.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:04.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:04.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:04.851 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:42:04.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:42:04.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:04.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:42:05.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:06.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:06.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:06.274 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:42:06.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:42:06.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:06.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:42:07.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:08.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:08.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:08.161 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:42:08.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:42:08.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:08.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:42:09.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:10.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:10.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:10.066 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:42:10.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:42:10.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:10.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:42:10.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:11.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:11.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:11.838 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:42:11.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:42:11.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:11.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:42:12.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:13.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:13.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:13.341 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-16T07:42:13.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T07:42:13.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:13.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:42:13.880 INFO:tasks.workunit.client.0.smithi162.stdout:osd.5 up in weight 1 up_from 51 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/4038945965,v1:127.0.0.1:6843/4038945965] [v2:127.0.0.1:6844/4038945965,v1:127.0.0.1:6845/4038945965] exists,up a0b3f08c-376b-45a6-822d-9ae441faac35 2024-09-16T07:42:13.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:42:13.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:42:13.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:42:13.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:42:13.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-16T07:42:13.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:42:13.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:42:13.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-09-16T07:42:13.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:42:13.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-16T07:42:13.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:42:13.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:42:13.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:42:13.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:42:13.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-16T07:42:13.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-16T07:42:13.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:42:13.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:42:13.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:42:13.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:42:13.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:42:13.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:42:13.886 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:42:13.886 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:42:13.886 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:42:13.886 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:42:13.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:42:13.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:42:13.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:42:13.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:42:13.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:42:13.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:42:13.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:42:13.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:42:13.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:42:13.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:42:13.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:42:13.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:42:13.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-16T07:42:13.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:42:13.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=61fd971b-d858-48d0-af9e-fa0e676e52d4 2024-09-16T07:42:13.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 61fd971b-d858-48d0-af9e-fa0e676e52d4' 2024-09-16T07:42:13.889 INFO:tasks.workunit.client.0.smithi162.stdout:add osd6 61fd971b-d858-48d0-af9e-fa0e676e52d4 2024-09-16T07:42:13.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:42:13.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBV4edmn0RYNhAAPgUu9Lzhp9hIBQ51gWrNMw== 2024-09-16T07:42:13.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBV4edmn0RYNhAAPgUu9Lzhp9hIBQ51gWrNMw=="}' 2024-09-16T07:42:13.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 61fd971b-d858-48d0-af9e-fa0e676e52d4 -i td/test-erasure-eio/6/new.json 2024-09-16T07:42:14.616 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:42:14.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-16T07:42:14.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBV4edmn0RYNhAAPgUu9Lzhp9hIBQ51gWrNMw== --osd-uuid 61fd971b-d858-48d0-af9e-fa0e676e52d4 2024-09-16T07:42:14.668 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:42:14.664+0000 7fe42a87f880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:42:14.726 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:42:14.724+0000 7fe42a87f880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:42:14.728 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:42:14.724+0000 7fe42a87f880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:42:14.730 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:42:14.724+0000 7fe42a87f880 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-16T07:42:14.730 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:42:14.724+0000 7fe42a87f880 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-16T07:42:14.730 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:42:14.724+0000 7fe42a87f880 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-16T07:42:14.880 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:42:14.876+0000 7fe42a87f880 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-16T07:42:21.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-16T07:42:21.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:42:21.361 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd6 key to auth repository 2024-09-16T07:42:21.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-09-16T07:42:21.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:42:21.946 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.6 2024-09-16T07:42:21.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-09-16T07:42:21.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:42:21.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:42:21.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:42:21.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:42:21.980 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:42:21.976+0000 7ff78765c880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:42:21.990 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:42:21.988+0000 7ff78765c880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:42:21.992 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:42:21.988+0000 7ff78765c880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:42:22.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-09-16T07:42:22.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:42:22.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-09-16T07:42:22.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:42:22.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:42:22.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:22.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:42:22.413 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:42:22.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:22.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:42:22.559 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:42:22.556+0000 7ff78765c880 -1 Falling back to public interface 2024-09-16T07:42:22.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:23.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:23.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:23.913 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:42:23.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:42:23.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:23.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:42:24.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:24.890 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:42:24.888+0000 7ff78765c880 -1 osd.6 0 log_to_monitors true 2024-09-16T07:42:25.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:25.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:25.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:42:25.360 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:42:25.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:25.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:42:25.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:26.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:26.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:26.971 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:42:26.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:42:26.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:26.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:42:27.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:28.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:28.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:28.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:42:28.644 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:42:28.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:28.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:42:29.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:30.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:30.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:30.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:42:30.138 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:42:30.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:42:30.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:30.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:31.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:31.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:31.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:42:31.712 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:42:31.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:31.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:42:32.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:33.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:33.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:33.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:42:33.302 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:42:33.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:33.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:42:33.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:34.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:34.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:34.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:42:34.828 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:42:34.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:34.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:42:35.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:36.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:36.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:36.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:42:36.284 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:42:36.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:36.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:42:36.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:37.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:37.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:37.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:42:37.735 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:42:37.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:37.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:42:38.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:39.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:39.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:39.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:42:39.237 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:42:39.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:39.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:42:39.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:40.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:40.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:40.876 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:42:40.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:42:40.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:40.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:42:41.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:42.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:42.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:42.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:42:42.670 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:42:42.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:42.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:42:43.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:44.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:44.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:44.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:42:44.575 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:42:44.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:44.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:42:45.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:46.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:46.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:46.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:42:46.313 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:42:46.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:46.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:42:46.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:47.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:47.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:47.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:42:47.778 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:42:47.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:47.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:42:48.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:49.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:49.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:49.770 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:42:49.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:42:49.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:49.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:42:50.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:42:51.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:42:51.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:42:51.322 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-16T07:42:51.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T07:42:51.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:42:51.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:42:51.820 INFO:tasks.workunit.client.0.smithi162.stdout:osd.6 up in weight 1 up_from 61 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/2883308578,v1:127.0.0.1:6851/2883308578] [v2:127.0.0.1:6852/2883308578,v1:127.0.0.1:6853/2883308578] exists,up 61fd971b-d858-48d0-af9e-fa0e676e52d4 2024-09-16T07:42:51.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:42:51.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:42:51.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:42:51.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-16T07:42:51.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T07:42:51.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T07:42:51.822 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:42:51.823 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:42:51.823 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:42:51.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.0.asok 2024-09-16T07:42:51.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-16T07:42:51.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.0.asok log flush 2024-09-16T07:42:51.910 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:42:51.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-16T07:42:51.929 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:38:39.045+0000 7f6de5b4b880 0 load: jerasure load: lrc 2024-09-16T07:42:51.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:364: TEST_ec_object_attr_read_error: local poolname=pool-jerasure 2024-09-16T07:42:51.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:365: TEST_ec_object_attr_read_error: create_erasure_coded_pool pool-jerasure 3 2 2024-09-16T07:42:51.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T07:42:51.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-16T07:42:51.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-16T07:42:51.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-16T07:42:51.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-16T07:42:51.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-16T07:42:51.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-16T07:42:53.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-16T07:42:53.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-16T07:42:54.227 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-16T07:42:54.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T07:42:55.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-16T07:42:55.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T07:42:55.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T07:42:55.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T07:42:55.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T07:42:55.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T07:42:55.242 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T07:42:55.242 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T07:42:55.243 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T07:42:55.243 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T07:42:55.243 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T07:42:55.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T07:42:55.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T07:42:55.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T07:42:55.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T07:42:55.462 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T07:42:55.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T07:42:55.960 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:42:55.960 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:42:55.960 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:42:55.960 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:42:55.960 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:42:55.960 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-16T07:42:55.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T07:42:55.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:42:55.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T07:42:56.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803837 2024-09-16T07:42:56.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803837 2024-09-16T07:42:56.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803837' 2024-09-16T07:42:56.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:42:56.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T07:42:56.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=55834574898 2024-09-16T07:42:56.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 55834574898 2024-09-16T07:42:56.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803837 1-55834574898' 2024-09-16T07:42:56.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:42:56.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T07:42:56.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378667 2024-09-16T07:42:56.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378667 2024-09-16T07:42:56.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803837 1-55834574898 2-81604378667' 2024-09-16T07:42:56.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:42:56.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T07:42:56.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=124554051621 2024-09-16T07:42:56.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 124554051621 2024-09-16T07:42:56.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803837 1-55834574898 2-81604378667 3-124554051621' 2024-09-16T07:42:56.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:42:56.574 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T07:42:56.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=171798691867 2024-09-16T07:42:56.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 171798691867 2024-09-16T07:42:56.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803837 1-55834574898 2-81604378667 3-124554051621 4-171798691867' 2024-09-16T07:42:56.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:42:56.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T07:42:56.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=219043332120 2024-09-16T07:42:56.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 219043332120 2024-09-16T07:42:56.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803837 1-55834574898 2-81604378667 3-124554051621 4-171798691867 5-219043332120' 2024-09-16T07:42:56.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:42:56.899 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-16T07:42:57.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=261993005068 2024-09-16T07:42:57.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 261993005068 2024-09-16T07:42:57.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803837 1-55834574898 2-81604378667 3-124554051621 4-171798691867 5-219043332120 6-261993005068' 2024-09-16T07:42:57.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:42:57.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803837 2024-09-16T07:42:57.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:42:57.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T07:42:57.058 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803837 2024-09-16T07:42:57.059 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:42:57.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803837 2024-09-16T07:42:57.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803837' 2024-09-16T07:42:57.061 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803837 2024-09-16T07:42:57.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:42:57.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803834 -lt 25769803837 2024-09-16T07:42:57.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T07:42:58.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T07:42:58.490 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:42:58.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803837 -lt 25769803837 2024-09-16T07:42:58.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:42:58.969 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-55834574898 2024-09-16T07:42:58.969 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:42:58.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T07:42:58.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-55834574898 2024-09-16T07:42:58.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:42:58.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=55834574898 2024-09-16T07:42:58.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 55834574898' 2024-09-16T07:42:58.975 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574898 2024-09-16T07:42:58.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T07:42:59.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 55834574899 -lt 55834574898 2024-09-16T07:42:59.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:42:59.430 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378667 2024-09-16T07:42:59.430 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:42:59.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T07:42:59.432 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378667 2024-09-16T07:42:59.432 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:42:59.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378667 2024-09-16T07:42:59.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378667' 2024-09-16T07:42:59.434 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 81604378667 2024-09-16T07:42:59.435 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T07:42:59.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378667 -lt 81604378667 2024-09-16T07:42:59.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:42:59.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-124554051621 2024-09-16T07:42:59.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:42:59.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T07:42:59.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-124554051621 2024-09-16T07:42:59.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:42:59.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=124554051621 2024-09-16T07:42:59.843 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 124554051621 2024-09-16T07:42:59.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 124554051621' 2024-09-16T07:42:59.844 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T07:43:00.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 124554051621 -lt 124554051621 2024-09-16T07:43:00.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:43:00.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-171798691867 2024-09-16T07:43:00.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:43:00.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T07:43:00.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-171798691867 2024-09-16T07:43:00.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:43:00.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=171798691867 2024-09-16T07:43:00.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 171798691867' 2024-09-16T07:43:00.331 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 171798691867 2024-09-16T07:43:00.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T07:43:00.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 171798691868 -lt 171798691867 2024-09-16T07:43:00.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:43:00.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-219043332120 2024-09-16T07:43:00.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:43:00.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T07:43:00.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-219043332120 2024-09-16T07:43:00.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:43:00.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=219043332120 2024-09-16T07:43:00.759 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 219043332120 2024-09-16T07:43:00.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 219043332120' 2024-09-16T07:43:00.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T07:43:01.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 219043332121 -lt 219043332120 2024-09-16T07:43:01.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:43:01.236 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 6-261993005068 2024-09-16T07:43:01.237 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:43:01.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=6 2024-09-16T07:43:01.240 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 6-261993005068 2024-09-16T07:43:01.240 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:43:01.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=261993005068 2024-09-16T07:43:01.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.6 seq 261993005068' 2024-09-16T07:43:01.242 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 261993005068 2024-09-16T07:43:01.243 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-16T07:43:01.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 261993005068 -lt 261993005068 2024-09-16T07:43:01.693 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T07:43:01.694 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:43:01.694 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:43:02.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T07:43:02.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T07:43:02.266 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T07:43:02.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T07:43:02.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T07:43:02.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T07:43:02.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T07:43:02.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T07:43:02.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T07:43:02.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T07:43:02.698 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:43:02.698 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:43:03.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T07:43:03.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T07:43:03.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T07:43:03.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:367: TEST_ec_object_attr_read_error: get_primary pool-jerasure myobject 2024-09-16T07:43:03.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=pool-jerasure 2024-09-16T07:43:03.213 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=myobject 2024-09-16T07:43:03.213 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map pool-jerasure myobject 2024-09-16T07:43:03.213 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T07:43:03.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:367: TEST_ec_object_attr_read_error: local primary_osd=3 2024-09-16T07:43:03.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:369: TEST_ec_object_attr_read_error: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-16T07:43:03.640 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:43:03.640 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:43:03.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:43:03.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:43:03.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:43:04.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:43:04.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:372: TEST_ec_object_attr_read_error: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-16T07:43:04.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-16T07:43:04.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-16T07:43:04.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-16T07:43:04.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:43:04.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-16T07:43:04.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:43:04.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-16T07:43:04.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:43:04.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-16T07:43:04.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:43:04.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-16T07:43:04.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-16T07:43:05.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:375: TEST_ec_object_attr_read_error: inject_eio ec mdata pool-jerasure myobject td/test-erasure-eio 1 2024-09-16T07:43:05.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=ec 2024-09-16T07:43:05.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T07:43:05.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=mdata 2024-09-16T07:43:05.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T07:43:05.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-jerasure 2024-09-16T07:43:05.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T07:43:05.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=myobject 2024-09-16T07:43:05.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T07:43:05.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/test-erasure-eio 2024-09-16T07:43:05.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T07:43:05.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=1 2024-09-16T07:43:05.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T07:43:05.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T07:43:05.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-jerasure myobject 2024-09-16T07:43:05.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T07:43:05.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-16T07:43:05.569 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-16T07:43:05.569 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:43:06.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2147483647 2024-09-16T07:43:06.061 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:43:06.061 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:43:06.061 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T07:43:06.061 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-16T07:43:06.062 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2147483647 5 1 0 6 2024-09-16T07:43:06.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T07:43:06.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=5 2024-09-16T07:43:06.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' ec '!=' ec ']' 2024-09-16T07:43:06.062 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/test-erasure-eio/5/type 2024-09-16T07:43:06.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T07:43:06.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 5 bluestore_debug_inject_read_err true 2024-09-16T07:43:06.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T07:43:06.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=5 2024-09-16T07:43:06.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T07:43:06.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T07:43:06.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T07:43:06.066 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.5 2024-09-16T07:43:06.066 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-16T07:43:06.066 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-16T07:43:06.067 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:43:06.067 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:43:06.067 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:43:06.067 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.5.asok 2024-09-16T07:43:06.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79228/ceph-osd.5.asok config set bluestore_debug_inject_read_err true 2024-09-16T07:43:06.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T07:43:06.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T07:43:06.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T07:43:06.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.5 2024-09-16T07:43:06.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-16T07:43:06.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-16T07:43:06.174 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:43:06.174 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:43:06.174 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:43:06.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.5.asok 2024-09-16T07:43:06.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T07:43:06.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.5.asok injectmdataerr pool-jerasure myobject 1 2024-09-16T07:43:06.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:378: TEST_ec_object_attr_read_error: activate_osd td/test-erasure-eio 3 2024-09-16T07:43:06.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-16T07:43:06.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T07:43:06.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-16T07:43:06.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T07:43:06.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-16T07:43:06.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:43:06.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:43:06.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:43:06.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:43:06.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-16T07:43:06.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-16T07:43:06.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T07:43:06.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T07:43:06.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:43:06.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T07:43:06.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:43:06.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:43:06.282 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:43:06.282 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:43:06.282 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:43:06.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:43:06.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:43:06.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:43:06.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:43:06.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:43:06.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:43:06.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:43:06.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:43:06.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:43:06.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T07:43:06.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T07:43:06.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-16T07:43:06.283 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-16T07:43:06.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-16T07:43:06.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:43:06.284 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-16T07:43:06.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-16T07:43:06.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T07:43:06.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T07:43:06.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T07:43:06.312 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:43:06.307+0000 7f871ace7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:43:06.331 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:43:06.327+0000 7f871ace7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:43:06.333 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:43:06.327+0000 7f871ace7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:43:06.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-16T07:43:06.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:43:06.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T07:43:06.766 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:43:06.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:43:06.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:43:06.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:43:06.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:43:06.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:43:06.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:43:06.897 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:43:06.891+0000 7f871ace7880 -1 Falling back to public interface 2024-09-16T07:43:07.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:43:08.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:43:08.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:43:08.261 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:43:08.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:43:08.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:43:08.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:43:08.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:43:09.743 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:43:09.739+0000 7f871ace7880 -1 osd.3 69 log_to_monitors true 2024-09-16T07:43:09.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:43:09.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:43:09.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:43:09.850 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:43:09.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:43:09.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:43:10.416 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 73 up_thru 67 down_at 70 last_clean_interval [29,69) [v2:127.0.0.1:6826/2311477087,v1:127.0.0.1:6827/2311477087] [v2:127.0.0.1:6828/2311477087,v1:127.0.0.1:6829/2311477087] exists,up 5619885f-c5d9-495f-953f-567d45139df6 2024-09-16T07:43:10.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:43:10.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:43:10.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:43:10.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:381: TEST_ec_object_attr_read_error: wait_for_clean 2024-09-16T07:43:10.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T07:43:10.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T07:43:10.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T07:43:10.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T07:43:10.418 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T07:43:10.419 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T07:43:10.419 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T07:43:10.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T07:43:10.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T07:43:10.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T07:43:10.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T07:43:10.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T07:43:10.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T07:43:10.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T07:43:10.629 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T07:43:11.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T07:43:11.079 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:43:11.080 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:43:11.080 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:43:11.080 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:43:11.080 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:43:11.080 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-16T07:43:11.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T07:43:11.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:43:11.080 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T07:43:11.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803842 2024-09-16T07:43:11.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803842 2024-09-16T07:43:11.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803842' 2024-09-16T07:43:11.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:43:11.266 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T07:43:11.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=55834574903 2024-09-16T07:43:11.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 55834574903 2024-09-16T07:43:11.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803842 1-55834574903' 2024-09-16T07:43:11.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:43:11.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T07:43:11.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378672 2024-09-16T07:43:11.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378672 2024-09-16T07:43:11.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803842 1-55834574903 2-81604378672' 2024-09-16T07:43:11.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:43:11.713 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T07:43:11.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=313532612611 2024-09-16T07:43:11.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 313532612611 2024-09-16T07:43:11.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803842 1-55834574903 2-81604378672 3-313532612611' 2024-09-16T07:43:11.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:43:11.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T07:43:12.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=171798691872 2024-09-16T07:43:12.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 171798691872 2024-09-16T07:43:12.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803842 1-55834574903 2-81604378672 3-313532612611 4-171798691872' 2024-09-16T07:43:12.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:43:12.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T07:43:12.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=219043332125 2024-09-16T07:43:12.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 219043332125 2024-09-16T07:43:12.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803842 1-55834574903 2-81604378672 3-313532612611 4-171798691872 5-219043332125' 2024-09-16T07:43:12.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:43:12.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-16T07:43:12.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=261993005073 2024-09-16T07:43:12.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 261993005073 2024-09-16T07:43:12.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803842 1-55834574903 2-81604378672 3-313532612611 4-171798691872 5-219043332125 6-261993005073' 2024-09-16T07:43:12.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:43:12.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:43:12.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803842 2024-09-16T07:43:12.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T07:43:12.321 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803842 2024-09-16T07:43:12.321 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:43:12.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803842 2024-09-16T07:43:12.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803842' 2024-09-16T07:43:12.323 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803842 2024-09-16T07:43:12.324 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:43:12.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803842 -lt 25769803842 2024-09-16T07:43:12.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:43:12.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-55834574903 2024-09-16T07:43:12.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:43:12.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T07:43:12.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-55834574903 2024-09-16T07:43:12.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:43:12.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=55834574903 2024-09-16T07:43:12.764 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574903 2024-09-16T07:43:12.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 55834574903' 2024-09-16T07:43:12.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T07:43:13.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 55834574903 -lt 55834574903 2024-09-16T07:43:13.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:43:13.213 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378672 2024-09-16T07:43:13.213 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:43:13.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T07:43:13.216 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378672 2024-09-16T07:43:13.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:43:13.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378672 2024-09-16T07:43:13.219 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 81604378672 2024-09-16T07:43:13.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378672' 2024-09-16T07:43:13.220 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T07:43:13.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378672 -lt 81604378672 2024-09-16T07:43:13.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:43:13.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-313532612611 2024-09-16T07:43:13.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:43:13.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T07:43:13.693 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-313532612611 2024-09-16T07:43:13.693 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:43:13.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=313532612611 2024-09-16T07:43:13.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 313532612611' 2024-09-16T07:43:13.696 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 313532612611 2024-09-16T07:43:13.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T07:43:14.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 313532612611 -lt 313532612611 2024-09-16T07:43:14.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:43:14.141 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-171798691872 2024-09-16T07:43:14.141 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:43:14.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T07:43:14.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-171798691872 2024-09-16T07:43:14.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:43:14.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=171798691872 2024-09-16T07:43:14.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 171798691872' 2024-09-16T07:43:14.146 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 171798691872 2024-09-16T07:43:14.146 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T07:43:14.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 171798691872 -lt 171798691872 2024-09-16T07:43:14.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:43:14.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-219043332125 2024-09-16T07:43:14.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:43:14.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T07:43:14.655 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-219043332125 2024-09-16T07:43:14.655 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:43:14.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=219043332125 2024-09-16T07:43:14.657 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 219043332125 2024-09-16T07:43:14.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 219043332125' 2024-09-16T07:43:14.658 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T07:43:15.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 219043332126 -lt 219043332125 2024-09-16T07:43:15.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:43:15.100 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 6-261993005073 2024-09-16T07:43:15.100 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:43:15.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=6 2024-09-16T07:43:15.103 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 6-261993005073 2024-09-16T07:43:15.104 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:43:15.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=261993005073 2024-09-16T07:43:15.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.6 seq 261993005073' 2024-09-16T07:43:15.105 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 261993005073 2024-09-16T07:43:15.106 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-16T07:43:15.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 261993005073 -lt 261993005073 2024-09-16T07:43:15.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T07:43:15.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:43:15.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:43:16.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T07:43:16.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T07:43:16.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T07:43:16.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T07:43:16.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T07:43:16.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T07:43:16.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T07:43:16.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T07:43:16.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T07:43:16.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T07:43:16.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:43:16.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:43:17.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T07:43:17.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T07:43:17.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T07:43:17.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:383: TEST_ec_object_attr_read_error: rados_get td/test-erasure-eio pool-jerasure myobject 2024-09-16T07:43:17.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T07:43:17.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T07:43:17.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-09-16T07:43:17.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-16T07:43:17.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-16T07:43:17.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-09-16T07:43:17.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-16T07:43:17.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-16T07:43:17.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:385: TEST_ec_object_attr_read_error: delete_erasure_coded_pool pool-jerasure 2024-09-16T07:43:17.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T07:43:17.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-16T07:43:17.803 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-16T07:43:17.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-16T07:43:18.585 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-16T07:43:18.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-16T07:43:18.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T07:43:18.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:43:18.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T07:43:18.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:43:18.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:43:18.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:43:18.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:43:18.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:43:18.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:43:18.919 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:43:18.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:43:18.921 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:43:18.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:43:18.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:43:18.924 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:43:18.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:43:18.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:43:18.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:43:18.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:43:18.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:43:18.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:43:18.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T07:43:18.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:43:18.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:43:18.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:43:18.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T07:43:18.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:43:18.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:43:18.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-16T07:43:18.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-16T07:43:18.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-16T07:43:18.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-16T07:43:18.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T07:43:18.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:43:18.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T07:43:18.974 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:43:18.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:43:18.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:43:18.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:43:18.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:43:18.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:43:18.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:43:18.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:43:18.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:43:18.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:43:18.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:43:18.985 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:43:18.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:43:18.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:43:18.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:43:18.988 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:43:18.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:43:18.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:43:18.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T07:43:18.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:43:18.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:43:18.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:43:18.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T07:43:18.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:43:18.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:43:18.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-16T07:43:18.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T07:43:18.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:43:18.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:43:18.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79228 2024-09-16T07:43:19.002 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T07:43:19.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T07:43:19.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T07:43:19.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-16T07:43:19.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-16T07:43:19.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-16T07:43:19.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T07:43:19.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T07:43:19.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T07:43:19.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-16T07:43:19.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-16T07:43:19.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T07:43:19.508 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:43:19.508 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:43:19.508 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:43:19.508 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:43:19.508 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:43:19.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:43:19.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T07:43:19.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T07:43:19.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T07:43:19.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:43:19.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:43:19.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T07:43:19.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T07:43:19.753 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:43:19.753 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:43:19.753 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:43:19.754 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:43:19.754 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:43:19.754 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:43:19.754 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T07:43:19.755 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:43:19.755 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get fsid 2024-09-16T07:43:19.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T07:43:19.851 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:43:19.851 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:43:19.851 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T07:43:19.851 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T07:43:19.852 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:43:19.852 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:43:19.852 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:43:19.852 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:43:19.852 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:43:19.853 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:43:19.853 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T07:43:19.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:43:19.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get mon_host 2024-09-16T07:43:19.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-16T07:43:19.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-16T07:43:19.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T07:43:19.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T07:43:19.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T07:43:19.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-16T07:43:19.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T07:43:20.862 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T07:43:20.862 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:43:20.862 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:43:20.863 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:43:20.863 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:43:20.863 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:43:20.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:43:20.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:43:20.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:43:20.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-16T07:43:20.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T07:43:21.252 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-16T07:43:21.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T07:43:22.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-16T07:43:22.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:43:22.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:43:22.264 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:43:22.264 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:43:22.265 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:43:22.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T07:43:22.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-16T07:43:22.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-mon.a.asok log flush 2024-09-16T07:43:22.349 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:43:22.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-16T07:43:22.362 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:43:19.559+0000 7f43f6e76d40 0 load: jerasure load: lrc 2024-09-16T07:43:22.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_errors td/test-erasure-eio 2024-09-16T07:43:22.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:418: TEST_ec_recovery_multiple_errors: local dir=td/test-erasure-eio 2024-09-16T07:43:22.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:419: TEST_ec_recovery_multiple_errors: local objname=myobject 2024-09-16T07:43:22.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:421: TEST_ec_recovery_multiple_errors: setup_osds 9 2024-09-16T07:43:22.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=9 2024-09-16T07:43:22.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-16T07:43:22.364 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 9 - 1 2024-09-16T07:43:22.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 8 2024-09-16T07:43:22.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:43:22.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-16T07:43:22.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:43:22.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:43:22.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T07:43:22.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:43:22.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-16T07:43:22.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:43:22.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:43:22.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:43:22.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:43:22.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-16T07:43:22.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-16T07:43:22.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:43:22.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:43:22.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:43:22.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:43:22.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:43:22.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:43:22.368 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:43:22.368 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:43:22.368 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:43:22.370 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:43:22.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:43:22.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:43:22.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:43:22.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:43:22.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:43:22.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:43:22.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:43:22.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:43:22.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:43:22.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:43:22.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:43:22.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:43:22.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-16T07:43:22.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:43:22.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6420d51f-7f82-42ca-90d4-468370c96868 2024-09-16T07:43:22.372 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 6420d51f-7f82-42ca-90d4-468370c96868 2024-09-16T07:43:22.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6420d51f-7f82-42ca-90d4-468370c96868' 2024-09-16T07:43:22.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:43:22.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCa4edm7Y2TFxAAJHGQr+qGLJyWNy3VA2WoDA== 2024-09-16T07:43:22.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCa4edm7Y2TFxAAJHGQr+qGLJyWNy3VA2WoDA=="}' 2024-09-16T07:43:22.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6420d51f-7f82-42ca-90d4-468370c96868 -i td/test-erasure-eio/0/new.json 2024-09-16T07:43:22.738 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:43:22.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-16T07:43:22.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCa4edm7Y2TFxAAJHGQr+qGLJyWNy3VA2WoDA== --osd-uuid 6420d51f-7f82-42ca-90d4-468370c96868 2024-09-16T07:43:22.791 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:43:22.787+0000 7fb08cf0d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:43:22.793 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:43:22.787+0000 7fb08cf0d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:43:22.795 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:43:22.791+0000 7fb08cf0d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:43:22.796 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:43:22.791+0000 7fb08cf0d880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T07:43:22.796 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:43:22.791+0000 7fb08cf0d880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T07:43:22.796 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:43:22.791+0000 7fb08cf0d880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T07:43:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:43:22.859+0000 7fb08cf0d880 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-16T07:43:28.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-16T07:43:28.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:43:28.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T07:43:28.605 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T07:43:28.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:43:29.176 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T07:43:29.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T07:43:29.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:43:29.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:43:29.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:43:29.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:43:29.211 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:43:29.207+0000 7f9cfb50e880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:43:29.219 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:43:29.215+0000 7f9cfb50e880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:43:29.220 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:43:29.215+0000 7f9cfb50e880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:43:29.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T07:43:29.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:43:29.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T07:43:29.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:43:29.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:43:29.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:43:29.608 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:43:29.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:43:29.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:43:29.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:43:29.768 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:43:29.763+0000 7f9cfb50e880 -1 Falling back to public interface 2024-09-16T07:43:30.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:43:31.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:43:31.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:43:31.004 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:43:31.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:43:31.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:43:31.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:43:31.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:43:32.094 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:43:32.091+0000 7f9cfb50e880 -1 osd.0 0 log_to_monitors true 2024-09-16T07:43:32.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:43:32.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:43:32.464 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:43:32.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:43:32.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:43:32.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:43:33.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:43:34.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:43:34.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:43:34.090 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:43:34.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:43:34.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:43:34.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:43:34.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:43:35.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:43:35.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:43:35.524 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:43:35.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:43:35.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:43:35.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:43:35.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:43:36.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:43:36.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:43:36.959 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:43:36.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:43:36.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:43:36.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:43:37.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:43:38.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:43:38.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:43:38.436 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:43:38.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:43:38.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:43:38.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:43:39.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:43:40.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:43:40.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:43:40.022 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:43:40.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:43:40.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:43:40.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:43:40.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:43:41.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:43:41.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:43:41.466 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:43:41.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:43:41.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:43:41.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:43:41.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:43:42.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:43:42.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:43:42.936 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:43:42.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:43:42.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:43:42.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:43:43.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:43:44.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:43:44.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:43:44.346 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:43:44.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:43:44.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:43:44.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:43:45.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:43:46.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:43:46.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:43:46.010 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:43:46.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:43:46.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:43:46.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:43:46.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:43:47.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:43:47.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:43:47.607 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:43:47.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:43:47.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:43:47.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:43:48.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:43:49.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:43:49.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:43:49.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:43:49.584 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:43:49.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:43:49.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:43:50.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:43:51.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:43:51.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:43:51.214 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:43:51.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:43:51.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:43:51.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:43:51.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:43:52.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:43:52.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:43:52.918 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:43:52.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:43:52.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:43:52.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:43:53.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:43:54.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:43:54.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:43:54.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:43:54.506 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:43:54.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:43:54.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:43:55.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:43:56.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:43:56.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:43:56.353 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:43:56.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:43:56.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:43:56.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:43:56.993 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/604360623,v1:127.0.0.1:6803/604360623] [v2:127.0.0.1:6804/604360623,v1:127.0.0.1:6805/604360623] exists,up 6420d51f-7f82-42ca-90d4-468370c96868 2024-09-16T07:43:56.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:43:56.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:43:56.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:43:56.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:43:56.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-16T07:43:56.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:43:56.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:43:56.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T07:43:56.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:43:56.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-16T07:43:56.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:43:56.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:43:56.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:43:56.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:43:56.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-16T07:43:57.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-16T07:43:57.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:43:57.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:43:57.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:43:57.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:43:57.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:43:57.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:43:57.000 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:43:57.000 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:43:57.000 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:43:57.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:43:57.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:43:57.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:43:57.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:43:57.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:43:57.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:43:57.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:43:57.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:43:57.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:43:57.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:43:57.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:43:57.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:43:57.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:43:57.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-16T07:43:57.002 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:43:57.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f00a11ff-637e-4390-a865-569444587b33 2024-09-16T07:43:57.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f00a11ff-637e-4390-a865-569444587b33' 2024-09-16T07:43:57.003 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 f00a11ff-637e-4390-a865-569444587b33 2024-09-16T07:43:57.003 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:43:57.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC94edmn7NVARAAISfT8nf/v55FF/K/ILRQjQ== 2024-09-16T07:43:57.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC94edmn7NVARAAISfT8nf/v55FF/K/ILRQjQ=="}' 2024-09-16T07:43:57.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f00a11ff-637e-4390-a865-569444587b33 -i td/test-erasure-eio/1/new.json 2024-09-16T07:43:57.574 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:43:57.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-16T07:43:57.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC94edmn7NVARAAISfT8nf/v55FF/K/ILRQjQ== --osd-uuid f00a11ff-637e-4390-a865-569444587b33 2024-09-16T07:43:57.627 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:43:57.626+0000 7efc0f078880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:43:57.655 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:43:57.654+0000 7efc0f078880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:43:57.657 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:43:57.654+0000 7efc0f078880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:43:57.658 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:43:57.654+0000 7efc0f078880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T07:43:57.658 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:43:57.654+0000 7efc0f078880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T07:43:57.658 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:43:57.654+0000 7efc0f078880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T07:43:57.788 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:43:57.786+0000 7efc0f078880 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-16T07:44:04.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-16T07:44:04.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:44:04.077 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-16T07:44:04.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T07:44:04.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:44:04.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T07:44:04.679 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-16T07:44:04.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:44:04.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:44:04.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:44:04.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:44:04.713 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:44:04.710+0000 7febaa750880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:44:04.715 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:44:04.714+0000 7febaa750880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:44:04.716 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:44:04.714+0000 7febaa750880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:44:05.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T07:44:05.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:44:05.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T07:44:05.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:44:05.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:44:05.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:05.155 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:44:05.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:44:05.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:05.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:44:05.296 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:44:05.294+0000 7febaa750880 -1 Falling back to public interface 2024-09-16T07:44:05.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:06.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:06.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:06.659 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:44:06.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:44:06.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:06.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:44:07.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:07.866 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:44:07.862+0000 7febaa750880 -1 osd.1 0 log_to_monitors true 2024-09-16T07:44:08.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:08.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:08.129 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:44:08.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:44:08.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:08.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:44:08.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:09.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:09.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:09.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:44:09.716 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:44:09.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:09.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:44:10.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:11.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:11.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:11.176 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:44:11.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:44:11.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:11.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:44:11.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:12.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:12.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:12.624 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:44:12.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:44:12.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:12.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:44:13.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:14.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:14.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:14.206 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:44:14.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:44:14.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:14.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:44:14.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:15.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:15.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:15.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:44:15.877 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:44:15.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:15.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:44:16.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:17.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:17.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:17.322 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:44:17.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:44:17.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:44:17.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:17.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:18.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:18.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:18.825 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:44:18.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:44:18.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:44:18.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:19.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:20.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:20.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:20.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:44:20.400 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:44:20.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:20.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:44:21.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:22.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:22.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:22.234 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:44:22.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:44:22.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:22.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:44:22.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:23.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:23.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:23.718 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:44:23.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:44:23.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:23.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:44:24.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:25.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:25.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:25.777 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:44:25.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:44:25.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:25.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:44:26.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:27.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:27.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:27.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:44:27.338 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:44:27.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:27.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:44:28.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:29.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:29.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:29.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:44:29.253 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:44:29.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:44:29.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:29.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:30.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:30.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:30.839 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:44:30.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:44:30.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:30.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:44:31.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:32.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:32.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:32.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:44:32.623 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:44:32.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:44:32.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:33.246 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2377158845,v1:127.0.0.1:6811/2377158845] [v2:127.0.0.1:6812/2377158845,v1:127.0.0.1:6813/2377158845] exists,up f00a11ff-637e-4390-a865-569444587b33 2024-09-16T07:44:33.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:44:33.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:44:33.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:44:33.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:44:33.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-16T07:44:33.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:44:33.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:44:33.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T07:44:33.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:44:33.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-16T07:44:33.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:44:33.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:44:33.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:44:33.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:44:33.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-16T07:44:33.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-16T07:44:33.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:44:33.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:44:33.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:44:33.252 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:44:33.252 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:44:33.252 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:44:33.252 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:44:33.252 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:44:33.252 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:44:33.252 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:44:33.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:44:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:44:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:44:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:44:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:44:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:44:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:44:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:44:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:44:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:44:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:44:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:44:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-16T07:44:33.254 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:44:33.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5a887ac4-a97c-4980-ac3a-7a10d6e04f1c 2024-09-16T07:44:33.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5a887ac4-a97c-4980-ac3a-7a10d6e04f1c' 2024-09-16T07:44:33.254 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 5a887ac4-a97c-4980-ac3a-7a10d6e04f1c 2024-09-16T07:44:33.255 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:44:33.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDh4edmPa9+EBAAOpfRTyH5mj3j5jNXeeLGOQ== 2024-09-16T07:44:33.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDh4edmPa9+EBAAOpfRTyH5mj3j5jNXeeLGOQ=="}' 2024-09-16T07:44:33.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5a887ac4-a97c-4980-ac3a-7a10d6e04f1c -i td/test-erasure-eio/2/new.json 2024-09-16T07:44:33.852 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:44:33.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-16T07:44:33.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDh4edmPa9+EBAAOpfRTyH5mj3j5jNXeeLGOQ== --osd-uuid 5a887ac4-a97c-4980-ac3a-7a10d6e04f1c 2024-09-16T07:44:33.910 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:44:33.905+0000 7f39167f7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:44:33.943 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:44:33.941+0000 7f39167f7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:44:33.945 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:44:33.941+0000 7f39167f7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:44:33.946 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:44:33.945+0000 7f39167f7880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T07:44:33.946 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:44:33.945+0000 7f39167f7880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T07:44:33.946 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:44:33.945+0000 7f39167f7880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T07:44:34.030 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:44:34.025+0000 7f39167f7880 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-16T07:44:40.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-16T07:44:40.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:44:40.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T07:44:40.346 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-16T07:44:40.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:44:40.951 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-16T07:44:40.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T07:44:40.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:44:40.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:44:40.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:44:40.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:44:40.985 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:44:40.981+0000 7f5e6a29f880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:44:40.988 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:44:40.985+0000 7f5e6a29f880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:44:40.990 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:44:40.985+0000 7f5e6a29f880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:44:41.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T07:44:41.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:44:41.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T07:44:41.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:44:41.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:44:41.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:41.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:44:41.405 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:44:41.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:44:41.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:41.568 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:44:41.565+0000 7f5e6a29f880 -1 Falling back to public interface 2024-09-16T07:44:41.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:42.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:42.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:42.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:44:42.801 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:44:42.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:42.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:44:43.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:44.157 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:44:44.153+0000 7f5e6a29f880 -1 osd.2 0 log_to_monitors true 2024-09-16T07:44:44.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:44.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:44.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:44:44.255 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:44:44.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:44:44.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:44.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:45.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:45.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:45.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:44:45.900 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:44:45.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:45.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:44:46.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:47.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:47.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:47.364 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:44:47.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:44:47.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:47.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:44:47.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:48.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:48.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:48.800 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:44:48.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:44:48.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:48.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:44:49.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:50.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:50.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:50.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:44:50.512 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:44:50.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:50.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:44:50.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:51.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:51.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:51.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:44:51.986 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:44:51.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:51.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:44:52.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:53.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:53.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:53.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:44:53.453 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:44:53.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:53.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:44:54.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:55.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:55.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:55.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:44:55.009 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:44:55.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:55.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:44:55.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:56.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:56.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:56.448 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:44:56.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:44:56.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:56.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:44:56.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:57.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:57.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:57.918 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:44:57.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:44:57.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:57.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:44:58.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:44:59.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:44:59.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:44:59.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:44:59.920 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:44:59.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:44:59.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:45:00.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:45:01.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:45:01.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:01.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:45:01.694 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:45:01.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:01.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:45:02.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:45:03.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:45:03.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:03.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:45:03.564 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:45:03.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:03.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:45:04.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:45:05.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:45:05.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:05.236 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:45:05.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:45:05.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:05.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:45:05.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:45:06.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:45:06.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:06.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:45:06.728 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:45:06.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:06.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:45:07.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:45:08.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:45:08.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:08.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:45:08.580 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:45:08.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:08.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:45:09.062 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2200345178,v1:127.0.0.1:6819/2200345178] [v2:127.0.0.1:6820/2200345178,v1:127.0.0.1:6821/2200345178] exists,up 5a887ac4-a97c-4980-ac3a-7a10d6e04f1c 2024-09-16T07:45:09.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:45:09.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:45:09.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:45:09.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:45:09.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-16T07:45:09.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:45:09.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:45:09.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T07:45:09.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:45:09.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-16T07:45:09.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:45:09.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:45:09.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:45:09.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:45:09.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-16T07:45:09.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-16T07:45:09.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:45:09.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:45:09.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:45:09.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:45:09.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:45:09.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:45:09.066 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:45:09.066 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:45:09.066 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:45:09.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:45:09.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:45:09.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:45:09.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:45:09.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:45:09.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:45:09.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:45:09.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:45:09.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:45:09.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:45:09.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:45:09.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:45:09.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:45:09.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-16T07:45:09.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:45:09.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=429b594d-9eee-4e65-9b90-cac8a5facc83 2024-09-16T07:45:09.069 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 429b594d-9eee-4e65-9b90-cac8a5facc83 2024-09-16T07:45:09.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 429b594d-9eee-4e65-9b90-cac8a5facc83' 2024-09-16T07:45:09.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:45:09.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAF4udmUDG5BRAAu9B52ZRyRAjHrJnAN0N55g== 2024-09-16T07:45:09.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAF4udmUDG5BRAAu9B52ZRyRAjHrJnAN0N55g=="}' 2024-09-16T07:45:09.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 429b594d-9eee-4e65-9b90-cac8a5facc83 -i td/test-erasure-eio/3/new.json 2024-09-16T07:45:10.137 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:45:10.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-16T07:45:10.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAF4udmUDG5BRAAu9B52ZRyRAjHrJnAN0N55g== --osd-uuid 429b594d-9eee-4e65-9b90-cac8a5facc83 2024-09-16T07:45:10.195 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:45:10.192+0000 7ff86b65b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:45:10.199 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:45:10.196+0000 7ff86b65b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:45:10.200 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:45:10.196+0000 7ff86b65b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:45:10.202 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:45:10.196+0000 7ff86b65b880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T07:45:10.202 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:45:10.196+0000 7ff86b65b880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T07:45:10.202 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:45:10.196+0000 7ff86b65b880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T07:45:10.281 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:45:10.276+0000 7ff86b65b880 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-16T07:45:16.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-16T07:45:16.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:45:16.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T07:45:16.661 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-16T07:45:16.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:45:17.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T07:45:17.268 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-16T07:45:17.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:45:17.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:45:17.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:45:17.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:45:17.304 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:45:17.300+0000 7fd505282880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:45:17.319 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:45:17.316+0000 7fd505282880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:45:17.321 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:45:17.316+0000 7fd505282880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:45:17.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T07:45:17.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:45:17.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T07:45:17.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:45:17.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:45:17.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:17.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:45:17.749 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:45:17.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:17.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:45:17.920 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:45:17.916+0000 7fd505282880 -1 Falling back to public interface 2024-09-16T07:45:18.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:45:19.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:45:19.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:19.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:45:19.267 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:45:19.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:19.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:45:19.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:45:20.587 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:45:20.584+0000 7fd505282880 -1 osd.3 0 log_to_monitors true 2024-09-16T07:45:20.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:45:20.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:20.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:45:20.696 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:45:20.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:20.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:45:21.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:45:22.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:45:22.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:22.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:45:22.191 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:45:22.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:22.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:45:23.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:45:24.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:45:24.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:24.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:45:24.202 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:45:24.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:24.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:45:24.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:45:25.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:45:25.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:25.870 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:45:25.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:45:25.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:25.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:45:26.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:45:27.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:45:27.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:27.424 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:45:27.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:45:27.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:27.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:45:27.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:45:28.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:45:28.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:28.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:45:28.969 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:45:28.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:28.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:45:29.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:45:30.390 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:45:30.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:45:30.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:30.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:45:30.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:30.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:45:30.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:45:31.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:45:31.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:31.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:45:31.888 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:45:31.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:31.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:45:32.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:45:33.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:45:33.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:33.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:45:33.450 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:45:33.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:33.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:45:33.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:45:34.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:45:34.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:34.981 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:45:34.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:45:34.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:34.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:45:35.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:45:36.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:45:36.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:36.969 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:45:36.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:45:36.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:36.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:45:37.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:45:38.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:45:38.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:38.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:45:38.796 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:45:38.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:38.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:45:39.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:45:40.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:45:40.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:40.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:45:40.304 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:45:40.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:40.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:45:41.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:45:42.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:45:42.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:42.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:45:42.212 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:45:42.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:42.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:45:43.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:45:44.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:45:44.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:44.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:45:44.469 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:45:44.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:44.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:45:45.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:45:46.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:45:46.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:46.435 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:45:46.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:45:46.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:46.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:45:46.930 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/302769405,v1:127.0.0.1:6827/302769405] [v2:127.0.0.1:6828/302769405,v1:127.0.0.1:6829/302769405] exists,up 429b594d-9eee-4e65-9b90-cac8a5facc83 2024-09-16T07:45:46.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:45:46.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:45:46.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:45:46.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:45:46.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-16T07:45:46.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:45:46.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:45:46.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-16T07:45:46.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:45:46.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-16T07:45:46.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:45:46.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:45:46.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:45:46.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:45:46.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-16T07:45:46.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-16T07:45:46.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:45:46.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:45:46.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:45:46.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:45:46.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:45:46.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:45:46.934 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:45:46.934 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:45:46.935 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:45:46.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:45:46.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:45:46.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:45:46.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:45:46.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:45:46.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:45:46.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:45:46.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:45:46.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:45:46.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:45:46.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:45:46.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:45:46.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:45:46.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-16T07:45:46.937 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:45:46.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ffe831fa-1394-4d86-b8fc-7012f7067378 2024-09-16T07:45:46.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 ffe831fa-1394-4d86-b8fc-7012f7067378' 2024-09-16T07:45:46.938 INFO:tasks.workunit.client.0.smithi162.stdout:add osd4 ffe831fa-1394-4d86-b8fc-7012f7067378 2024-09-16T07:45:46.938 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:45:46.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAq4udmDNxPORAAoFPKGXZZq66u0oL0QHFu5A== 2024-09-16T07:45:46.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAq4udmDNxPORAAoFPKGXZZq66u0oL0QHFu5A=="}' 2024-09-16T07:45:46.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ffe831fa-1394-4d86-b8fc-7012f7067378 -i td/test-erasure-eio/4/new.json 2024-09-16T07:45:48.404 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:45:48.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-16T07:45:48.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAq4udmDNxPORAAoFPKGXZZq66u0oL0QHFu5A== --osd-uuid ffe831fa-1394-4d86-b8fc-7012f7067378 2024-09-16T07:45:48.463 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:45:48.460+0000 7f28745ae880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:45:48.475 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:45:48.472+0000 7f28745ae880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:45:48.477 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:45:48.472+0000 7f28745ae880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:45:48.478 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:45:48.476+0000 7f28745ae880 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-16T07:45:48.478 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:45:48.476+0000 7f28745ae880 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-16T07:45:48.479 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:45:48.476+0000 7f28745ae880 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-16T07:45:48.617 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:45:48.612+0000 7f28745ae880 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-16T07:45:55.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-16T07:45:55.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:45:55.345 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd4 key to auth repository 2024-09-16T07:45:55.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-16T07:45:55.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:45:55.927 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.4 2024-09-16T07:45:55.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-16T07:45:55.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:45:55.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:45:55.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:45:55.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:45:55.961 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:45:55.955+0000 7f2b8b928880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:45:55.986 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:45:55.979+0000 7f2b8b928880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:45:55.987 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:45:55.983+0000 7f2b8b928880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:45:56.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-16T07:45:56.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:45:56.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-16T07:45:56.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:45:56.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:45:56.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:56.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:45:56.362 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:45:56.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:56.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:45:56.561 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:45:56.555+0000 7f2b8b928880 -1 Falling back to public interface 2024-09-16T07:45:56.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:45:57.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:45:57.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:57.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:45:57.815 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:45:57.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:57.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:45:58.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:45:59.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:45:59.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:45:59.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:45:59.228 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:45:59.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:45:59.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:45:59.229 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:45:59.223+0000 7f2b8b928880 -1 osd.4 0 log_to_monitors true 2024-09-16T07:45:59.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:00.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:00.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:00.714 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:46:00.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:46:00.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:00.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:46:01.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:02.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:02.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:02.407 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:46:02.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:46:02.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:02.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:46:03.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:04.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:04.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:04.025 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:46:04.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:46:04.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:04.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:46:04.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:05.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:05.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:05.488 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:46:05.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:46:05.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:05.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:46:06.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:07.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:07.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:07.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:46:07.052 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:46:07.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:07.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:46:07.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:08.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:08.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:08.543 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:46:08.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:46:08.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:08.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:46:09.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:10.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:10.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:10.070 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:46:10.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:46:10.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:10.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:46:10.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:11.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:11.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:11.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:46:11.484 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:46:11.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:11.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:46:12.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:13.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:13.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:13.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:46:13.068 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:46:13.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:13.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:46:13.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:14.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:14.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:14.600 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:46:14.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:46:14.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:14.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:46:15.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:16.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:16.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:16.528 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:46:16.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:46:16.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:16.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:46:17.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:18.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:18.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:18.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:46:18.548 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:46:18.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:18.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:46:19.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:20.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:20.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:20.330 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:46:20.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:46:20.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:20.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:46:21.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:22.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:22.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:22.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:46:22.332 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:46:22.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:22.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:46:23.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:24.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:24.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:24.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:46:24.060 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:46:24.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:24.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:46:24.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:25.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:25.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:25.845 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-16T07:46:25.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T07:46:25.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:25.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:46:26.576 INFO:tasks.workunit.client.0.smithi162.stdout:osd.4 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2087971596,v1:127.0.0.1:6835/2087971596] [v2:127.0.0.1:6836/2087971596,v1:127.0.0.1:6837/2087971596] exists,up ffe831fa-1394-4d86-b8fc-7012f7067378 2024-09-16T07:46:26.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:46:26.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:46:26.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:46:26.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:46:26.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-16T07:46:26.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:46:26.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:46:26.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-16T07:46:26.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:46:26.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-16T07:46:26.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:46:26.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:46:26.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:46:26.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:46:26.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-16T07:46:26.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-16T07:46:26.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:46:26.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:46:26.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:46:26.580 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:46:26.580 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:46:26.580 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:46:26.582 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:46:26.582 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:46:26.582 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:46:26.582 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:46:26.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:46:26.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:46:26.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:46:26.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:46:26.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:46:26.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:46:26.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:46:26.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:46:26.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:46:26.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:46:26.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:46:26.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:46:26.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-16T07:46:26.584 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:46:26.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c5e7fa6a-ed81-4e07-8ee5-4247bb6bd93c 2024-09-16T07:46:26.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 c5e7fa6a-ed81-4e07-8ee5-4247bb6bd93c' 2024-09-16T07:46:26.586 INFO:tasks.workunit.client.0.smithi162.stdout:add osd5 c5e7fa6a-ed81-4e07-8ee5-4247bb6bd93c 2024-09-16T07:46:26.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:46:26.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBS4udm1KSAJBAAdDxnRJrNwwKkGqW0/+fA0A== 2024-09-16T07:46:26.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBS4udm1KSAJBAAdDxnRJrNwwKkGqW0/+fA0A=="}' 2024-09-16T07:46:26.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c5e7fa6a-ed81-4e07-8ee5-4247bb6bd93c -i td/test-erasure-eio/5/new.json 2024-09-16T07:46:27.687 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:46:27.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-16T07:46:27.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBS4udm1KSAJBAAdDxnRJrNwwKkGqW0/+fA0A== --osd-uuid c5e7fa6a-ed81-4e07-8ee5-4247bb6bd93c 2024-09-16T07:46:27.738 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:46:27.735+0000 7f4499440880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:46:27.887 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:46:27.883+0000 7f4499440880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:46:27.889 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:46:27.883+0000 7f4499440880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:46:27.890 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:46:27.887+0000 7f4499440880 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-16T07:46:27.890 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:46:27.887+0000 7f4499440880 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-16T07:46:27.890 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:46:27.887+0000 7f4499440880 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-16T07:46:28.064 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:46:28.063+0000 7f4499440880 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-16T07:46:34.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-16T07:46:34.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:46:34.509 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd5 key to auth repository 2024-09-16T07:46:34.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-16T07:46:34.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:46:35.072 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.5 2024-09-16T07:46:35.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-16T07:46:35.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:46:35.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:46:35.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:46:35.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:46:35.105 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:46:35.099+0000 7fa9f26d8880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:46:35.143 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:46:35.139+0000 7fa9f26d8880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:46:35.144 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:46:35.139+0000 7fa9f26d8880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:46:35.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-16T07:46:35.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:46:35.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-16T07:46:35.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:46:35.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:46:35.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:35.575 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:46:35.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:46:35.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:35.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:46:35.696 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:46:35.690+0000 7fa9f26d8880 -1 Falling back to public interface 2024-09-16T07:46:36.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:37.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:37.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:37.067 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:46:37.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:46:37.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:37.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:46:37.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:38.314 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:46:38.306+0000 7fa9f26d8880 -1 osd.5 0 log_to_monitors true 2024-09-16T07:46:38.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:38.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:38.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:46:38.546 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:46:38.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:38.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:46:39.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:40.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:40.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:40.188 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:46:40.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:46:40.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:40.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:46:40.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:41.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:41.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:41.707 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:46:41.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:46:41.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:41.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:46:42.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:43.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:43.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:43.262 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:46:43.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:46:43.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:43.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:46:43.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:44.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:44.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:44.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:46:44.828 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:46:44.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:44.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:46:45.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:46.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:46.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:46.549 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:46:46.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:46:46.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:46.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:46:47.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:48.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:48.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:48.111 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:46:48.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:46:48.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:48.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:46:48.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:49.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:49.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:49.549 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:46:49.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:46:49.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:49.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:46:50.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:51.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:51.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:51.146 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:46:51.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:46:51.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:51.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:46:51.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:52.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:52.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:52.706 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:46:52.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:46:52.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:52.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:46:53.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:54.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:54.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:54.636 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:46:54.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:46:54.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:54.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:46:55.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:56.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:56.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:56.335 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:46:56.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:46:56.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:56.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:46:57.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:46:58.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:46:58.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:46:58.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:46:58.311 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:46:58.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:46:58.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:46:59.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:00.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:00.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:00.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:47:00.256 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:47:00.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:00.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:47:00.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:01.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:01.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:01.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:47:01.986 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:47:01.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:01.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:47:03.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:04.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:04.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:04.113 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:47:04.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:47:04.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:04.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:47:04.610 INFO:tasks.workunit.client.0.smithi162.stdout:osd.5 up in weight 1 up_from 51 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2590589598,v1:127.0.0.1:6843/2590589598] [v2:127.0.0.1:6844/2590589598,v1:127.0.0.1:6845/2590589598] exists,up c5e7fa6a-ed81-4e07-8ee5-4247bb6bd93c 2024-09-16T07:47:04.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:47:04.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:47:04.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:47:04.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:47:04.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-16T07:47:04.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:47:04.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:47:04.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-09-16T07:47:04.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:47:04.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-16T07:47:04.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:47:04.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:47:04.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:47:04.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:47:04.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-16T07:47:04.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-16T07:47:04.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:47:04.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:47:04.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:47:04.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:47:04.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:47:04.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:47:04.615 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:47:04.615 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:47:04.615 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:47:04.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:47:04.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:47:04.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:47:04.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:47:04.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:47:04.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:47:04.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:47:04.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:47:04.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:47:04.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:47:04.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:47:04.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:47:04.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:47:04.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-16T07:47:04.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:47:04.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c2de07d0-3029-452e-878b-bfebdced31f9 2024-09-16T07:47:04.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 c2de07d0-3029-452e-878b-bfebdced31f9' 2024-09-16T07:47:04.619 INFO:tasks.workunit.client.0.smithi162.stdout:add osd6 c2de07d0-3029-452e-878b-bfebdced31f9 2024-09-16T07:47:04.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:47:04.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB44udmP3STJhAA73fWl7OAGiX9wzoXS+r+XQ== 2024-09-16T07:47:04.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB44udmP3STJhAA73fWl7OAGiX9wzoXS+r+XQ=="}' 2024-09-16T07:47:04.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c2de07d0-3029-452e-878b-bfebdced31f9 -i td/test-erasure-eio/6/new.json 2024-09-16T07:47:05.900 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:47:05.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-16T07:47:05.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB44udmP3STJhAA73fWl7OAGiX9wzoXS+r+XQ== --osd-uuid c2de07d0-3029-452e-878b-bfebdced31f9 2024-09-16T07:47:05.958 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:47:05.954+0000 7f270ad27880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:47:05.994 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:47:05.990+0000 7f270ad27880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:47:05.996 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:47:05.994+0000 7f270ad27880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:47:05.997 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:47:05.994+0000 7f270ad27880 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-16T07:47:05.997 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:47:05.994+0000 7f270ad27880 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-16T07:47:05.998 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:47:05.994+0000 7f270ad27880 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-16T07:47:06.092 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:47:06.090+0000 7f270ad27880 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-16T07:47:12.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-16T07:47:12.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:47:12.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-09-16T07:47:12.469 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd6 key to auth repository 2024-09-16T07:47:12.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:47:13.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-09-16T07:47:13.030 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.6 2024-09-16T07:47:13.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:47:13.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:47:13.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:47:13.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:47:13.065 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:47:13.062+0000 7fc6b1cf9880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:47:13.101 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:47:13.098+0000 7fc6b1cf9880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:47:13.102 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:47:13.102+0000 7fc6b1cf9880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:47:13.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-09-16T07:47:13.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:47:13.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-09-16T07:47:13.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:47:13.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:47:13.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:13.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:47:13.479 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:47:13.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:13.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:47:13.665 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:47:13.662+0000 7fc6b1cf9880 -1 Falling back to public interface 2024-09-16T07:47:13.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:14.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:14.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:14.937 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:47:14.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:47:14.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:14.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:47:15.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:16.127 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:47:16.126+0000 7fc6b1cf9880 -1 osd.6 0 log_to_monitors true 2024-09-16T07:47:16.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:16.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:16.361 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:47:16.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:47:16.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:16.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:47:16.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:17.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:17.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:17.814 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:47:17.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:47:17.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:17.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:47:18.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:19.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:19.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:19.279 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:47:19.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:47:19.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:19.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:47:20.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:21.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:21.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:21.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:47:21.069 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:47:21.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:47:21.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:21.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:22.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:22.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:22.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:47:22.554 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:47:22.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:22.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:47:23.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:24.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:24.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:24.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:47:24.142 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:47:24.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:24.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:47:24.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:25.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:25.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:25.678 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:47:25.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:47:25.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:25.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:47:26.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:27.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:27.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:27.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:47:27.097 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:47:27.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:27.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:47:27.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:28.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:28.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:28.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:47:28.530 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:47:28.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:28.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:47:28.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:29.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:29.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:29.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:47:29.973 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:47:29.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:29.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:47:30.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:31.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:31.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:31.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:47:31.676 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:47:31.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:31.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:47:32.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:33.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:33.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:33.981 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:47:33.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:47:33.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:33.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:47:35.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:36.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:36.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:36.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:47:36.008 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:47:36.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:36.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:47:37.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:38.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:38.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:38.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:47:38.084 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:47:38.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:38.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:47:39.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:40.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:40.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:40.073 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:47:40.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:47:40.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:40.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:47:40.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:41.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:41.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:41.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:47:41.891 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:47:41.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:41.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:47:42.487 INFO:tasks.workunit.client.0.smithi162.stdout:osd.6 up in weight 1 up_from 60 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1478621526,v1:127.0.0.1:6851/1478621526] [v2:127.0.0.1:6852/1478621526,v1:127.0.0.1:6853/1478621526] exists,up c2de07d0-3029-452e-878b-bfebdced31f9 2024-09-16T07:47:42.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:47:42.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:47:42.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:47:42.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:47:42.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 7 2024-09-16T07:47:42.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:47:42.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:47:42.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-09-16T07:47:42.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:47:42.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/7 2024-09-16T07:47:42.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:47:42.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:47:42.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:47:42.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:47:42.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/7' 2024-09-16T07:47:42.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/7/journal' 2024-09-16T07:47:42.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:47:42.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:47:42.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:47:42.491 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:47:42.491 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:47:42.491 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:47:42.491 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:47:42.491 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:47:42.491 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:47:42.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:47:42.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:47:42.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:47:42.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:47:42.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:47:42.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:47:42.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:47:42.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:47:42.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:47:42.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:47:42.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:47:42.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:47:42.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:47:42.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/7 2024-09-16T07:47:42.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:47:42.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=31015ff9-0ba5-4d93-b702-3c0198d0908e 2024-09-16T07:47:42.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 31015ff9-0ba5-4d93-b702-3c0198d0908e' 2024-09-16T07:47:42.495 INFO:tasks.workunit.client.0.smithi162.stdout:add osd7 31015ff9-0ba5-4d93-b702-3c0198d0908e 2024-09-16T07:47:42.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:47:42.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCe4udmVgPPHhAA+ey95Ug1EFYv4vqQ3b3ECw== 2024-09-16T07:47:42.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCe4udmVgPPHhAA+ey95Ug1EFYv4vqQ3b3ECw=="}' 2024-09-16T07:47:42.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 31015ff9-0ba5-4d93-b702-3c0198d0908e -i td/test-erasure-eio/7/new.json 2024-09-16T07:47:43.540 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:47:43.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/7/new.json 2024-09-16T07:47:43.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCe4udmVgPPHhAA+ey95Ug1EFYv4vqQ3b3ECw== --osd-uuid 31015ff9-0ba5-4d93-b702-3c0198d0908e 2024-09-16T07:47:43.600 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:47:43.597+0000 7f699db86880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:47:43.629 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:47:43.625+0000 7f699db86880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:47:43.631 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:47:43.629+0000 7f699db86880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:47:43.631 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:47:43.629+0000 7f699db86880 -1 bluestore(td/test-erasure-eio/7/block) _read_bdev_label failed to open td/test-erasure-eio/7/block: (2) No such file or directory 2024-09-16T07:47:43.631 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:47:43.629+0000 7f699db86880 -1 bluestore(td/test-erasure-eio/7/block) _read_bdev_label failed to open td/test-erasure-eio/7/block: (2) No such file or directory 2024-09-16T07:47:43.631 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:47:43.629+0000 7f699db86880 -1 bluestore(td/test-erasure-eio/7/block) _read_bdev_label failed to open td/test-erasure-eio/7/block: (2) No such file or directory 2024-09-16T07:47:43.730 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:47:43.725+0000 7f699db86880 -1 bluestore(td/test-erasure-eio/7) _read_fsid unparsable uuid 2024-09-16T07:47:50.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/7/keyring 2024-09-16T07:47:50.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:47:50.669 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd7 key to auth repository 2024-09-16T07:47:50.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-09-16T07:47:50.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:47:51.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-09-16T07:47:51.326 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.7 2024-09-16T07:47:51.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:47:51.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:47:51.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:47:51.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:47:51.349 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:47:51.345+0000 7f07a6d3b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:47:51.351 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:47:51.349+0000 7f07a6d3b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:47:51.352 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:47:51.349+0000 7f07a6d3b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:47:51.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-09-16T07:47:51.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:47:51.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-09-16T07:47:51.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:47:51.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:47:51.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:51.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:47:51.785 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:47:51.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:51.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:47:51.929 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:47:51.925+0000 7f07a6d3b880 -1 Falling back to public interface 2024-09-16T07:47:52.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:53.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:53.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:53.255 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:47:53.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:47:53.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:53.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:47:53.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:54.516 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:47:54.513+0000 7f07a6d3b880 -1 osd.7 0 log_to_monitors true 2024-09-16T07:47:54.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:54.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:54.675 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:47:54.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:47:54.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:54.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:47:55.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:56.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:56.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:56.233 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:47:56.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:47:56.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:56.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:47:57.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:58.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:58.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:58.138 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:47:58.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:47:58.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:58.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:47:58.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:47:59.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:47:59.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:47:59.574 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:47:59.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:47:59.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:47:59.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:48:00.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:01.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:01.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:01.003 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:48:01.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:48:01.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:01.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:48:01.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:02.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:02.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:02.563 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:48:02.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:48:02.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:02.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:48:03.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:04.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:04.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:04.095 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:48:04.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:48:04.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:04.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:48:04.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:05.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:05.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:05.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:48:05.689 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:48:05.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:05.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:48:06.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:07.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:07.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:07.134 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:48:07.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:48:07.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:07.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:48:07.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:08.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:08.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:08.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:48:08.653 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:48:08.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:08.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:48:09.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:10.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:10.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:10.085 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:48:10.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:48:10.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:10.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:48:10.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:11.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:11.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:11.709 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:48:11.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:48:11.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:11.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:48:12.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:13.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:13.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:13.923 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:48:13.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:48:13.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:13.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:48:14.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:16.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:16.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:16.000 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:48:16.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:48:16.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:16.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:48:16.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:17.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:17.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:17.673 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:48:17.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:48:17.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:17.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:48:19.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:20.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:20.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:20.056 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:48:20.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:48:20.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:20.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:48:20.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:21.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:21.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:21.537 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-16T07:48:21.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T07:48:21.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:21.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-16T07:48:22.255 INFO:tasks.workunit.client.0.smithi162.stdout:osd.7 up in weight 1 up_from 71 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/3108266858,v1:127.0.0.1:6859/3108266858] [v2:127.0.0.1:6860/3108266858,v1:127.0.0.1:6861/3108266858] exists,up 31015ff9-0ba5-4d93-b702-3c0198d0908e 2024-09-16T07:48:22.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:48:22.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:48:22.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:48:22.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:48:22.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 8 2024-09-16T07:48:22.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:48:22.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:48:22.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-09-16T07:48:22.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:48:22.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/8 2024-09-16T07:48:22.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:48:22.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:48:22.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:48:22.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:48:22.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/8' 2024-09-16T07:48:22.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/8/journal' 2024-09-16T07:48:22.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:48:22.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:48:22.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:48:22.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:48:22.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:48:22.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:48:22.260 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:48:22.260 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:48:22.260 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:48:22.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:48:22.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:48:22.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:48:22.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:48:22.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:48:22.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:48:22.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:48:22.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:48:22.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:48:22.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:48:22.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:48:22.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:48:22.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:48:22.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/8 2024-09-16T07:48:22.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:48:22.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=16fec739-f771-4f5d-b7f6-7caeb9ee899f 2024-09-16T07:48:22.264 INFO:tasks.workunit.client.0.smithi162.stdout:add osd8 16fec739-f771-4f5d-b7f6-7caeb9ee899f 2024-09-16T07:48:22.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 16fec739-f771-4f5d-b7f6-7caeb9ee899f' 2024-09-16T07:48:22.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:48:22.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDG4udmbGxVERAA9fprBf+kNEpPYVjyhFIkaw== 2024-09-16T07:48:22.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDG4udmbGxVERAA9fprBf+kNEpPYVjyhFIkaw=="}' 2024-09-16T07:48:22.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 16fec739-f771-4f5d-b7f6-7caeb9ee899f -i td/test-erasure-eio/8/new.json 2024-09-16T07:48:23.229 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:48:23.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/8/new.json 2024-09-16T07:48:23.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDG4udmbGxVERAA9fprBf+kNEpPYVjyhFIkaw== --osd-uuid 16fec739-f771-4f5d-b7f6-7caeb9ee899f 2024-09-16T07:48:23.272 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:48:23.268+0000 7f9e202c7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:48:23.449 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:48:23.444+0000 7f9e202c7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:48:23.451 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:48:23.448+0000 7f9e202c7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:48:23.453 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:48:23.448+0000 7f9e202c7880 -1 bluestore(td/test-erasure-eio/8/block) _read_bdev_label failed to open td/test-erasure-eio/8/block: (2) No such file or directory 2024-09-16T07:48:23.453 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:48:23.448+0000 7f9e202c7880 -1 bluestore(td/test-erasure-eio/8/block) _read_bdev_label failed to open td/test-erasure-eio/8/block: (2) No such file or directory 2024-09-16T07:48:23.453 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:48:23.448+0000 7f9e202c7880 -1 bluestore(td/test-erasure-eio/8/block) _read_bdev_label failed to open td/test-erasure-eio/8/block: (2) No such file or directory 2024-09-16T07:48:23.636 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:48:23.632+0000 7f9e202c7880 -1 bluestore(td/test-erasure-eio/8) _read_fsid unparsable uuid 2024-09-16T07:48:29.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/8/keyring 2024-09-16T07:48:29.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:48:29.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-09-16T07:48:29.854 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd8 key to auth repository 2024-09-16T07:48:29.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:48:30.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-09-16T07:48:30.415 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.8 2024-09-16T07:48:30.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:48:30.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:48:30.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:48:30.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:48:30.444 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:48:30.440+0000 7f389d95d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:48:30.446 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:48:30.444+0000 7f389d95d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:48:30.448 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:48:30.444+0000 7f389d95d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:48:30.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-09-16T07:48:30.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:48:30.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-09-16T07:48:30.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:48:30.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:48:30.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:30.888 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:48:30.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:48:30.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:30.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:48:31.013 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:48:31.008+0000 7f389d95d880 -1 Falling back to public interface 2024-09-16T07:48:31.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:32.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:32.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:32.332 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:48:32.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:48:32.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:32.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:48:32.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:33.646 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:48:33.640+0000 7f389d95d880 -1 osd.8 0 log_to_monitors true 2024-09-16T07:48:33.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:33.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:33.811 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:48:33.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:48:33.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:33.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:48:34.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:35.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:35.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:35.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:48:35.714 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:48:35.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:35.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:48:36.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:37.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:37.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:37.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:48:37.223 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:48:37.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:37.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:48:37.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:38.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:38.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:38.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:48:38.815 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:48:38.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:38.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:48:39.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:40.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:40.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:40.348 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:48:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:48:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:48:40.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:41.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:41.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:41.997 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:48:41.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:48:41.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:41.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:48:42.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:43.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:43.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:43.646 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:48:43.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:48:43.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:43.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:48:44.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:45.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:45.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:45.176 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:48:45.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:48:45.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:45.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:48:45.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:46.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:46.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:46.753 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:48:46.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:48:46.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:46.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:48:47.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:48.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:48.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:48.332 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:48:48.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:48:48.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:48.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:48:49.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:50.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:50.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:50.191 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:48:50.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:48:50.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:50.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:48:51.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:52.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:52.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:52.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:48:52.537 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:48:52.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:52.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:48:53.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:54.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:54.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:54.431 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:48:54.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:48:54.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:54.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:48:55.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:56.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:56.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:56.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:48:56.284 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:48:56.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:56.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:48:57.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:48:58.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:48:58.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:48:58.642 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:48:58.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:48:58.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:48:58.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:48:59.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:49:00.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:49:00.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:49:00.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:49:00.231 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:49:00.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:49:00.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-16T07:49:01.011 INFO:tasks.workunit.client.0.smithi162.stdout:osd.8 up in weight 1 up_from 82 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/1085467272,v1:127.0.0.1:6867/1085467272] [v2:127.0.0.1:6868/1085467272,v1:127.0.0.1:6869/1085467272] exists,up 16fec739-f771-4f5d-b7f6-7caeb9ee899f 2024-09-16T07:49:01.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:49:01.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:49:01.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:49:01.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-16T07:49:01.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T07:49:01.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T07:49:01.013 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:49:01.013 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:49:01.013 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:49:01.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.0.asok 2024-09-16T07:49:01.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-16T07:49:01.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.0.asok log flush 2024-09-16T07:49:01.104 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:49:01.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-16T07:49:01.124 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:43:29.775+0000 7f9cfb50e880 0 load: jerasure load: lrc 2024-09-16T07:49:01.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:423: TEST_ec_recovery_multiple_errors: local poolname=pool-jerasure 2024-09-16T07:49:01.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:424: TEST_ec_recovery_multiple_errors: create_erasure_coded_pool pool-jerasure 4 4 2024-09-16T07:49:01.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T07:49:01.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-16T07:49:01.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=4 2024-09-16T07:49:01.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-16T07:49:01.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=4 2024-09-16T07:49:01.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-16T07:49:01.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=4 m=4 crush-failure-domain=osd 2024-09-16T07:49:01.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-16T07:49:01.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-16T07:49:03.367 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-16T07:49:03.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T07:49:04.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-16T07:49:04.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T07:49:04.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T07:49:04.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T07:49:04.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T07:49:04.380 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T07:49:04.380 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T07:49:04.381 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T07:49:04.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T07:49:04.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T07:49:04.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T07:49:04.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T07:49:04.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T07:49:04.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T07:49:04.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T07:49:04.588 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T07:49:05.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T07:49:05.220 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:49:05.220 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:49:05.220 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:49:05.220 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:49:05.220 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:49:05.220 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-16T07:49:05.220 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-16T07:49:05.220 INFO:tasks.workunit.client.0.smithi162.stderr:8' 2024-09-16T07:49:05.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T07:49:05.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:49:05.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T07:49:05.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803853 2024-09-16T07:49:05.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803853 2024-09-16T07:49:05.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803853' 2024-09-16T07:49:05.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:49:05.510 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T07:49:05.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=55834574914 2024-09-16T07:49:05.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 55834574914 2024-09-16T07:49:05.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803853 1-55834574914' 2024-09-16T07:49:05.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:49:05.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T07:49:05.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378688 2024-09-16T07:49:05.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378688 2024-09-16T07:49:05.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803853 1-55834574914 2-81604378688' 2024-09-16T07:49:05.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:49:05.845 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T07:49:06.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=124554051640 2024-09-16T07:49:06.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 124554051640 2024-09-16T07:49:06.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803853 1-55834574914 2-81604378688 3-124554051640' 2024-09-16T07:49:06.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:49:06.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T07:49:06.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=171798691887 2024-09-16T07:49:06.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 171798691887 2024-09-16T07:49:06.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803853 1-55834574914 2-81604378688 3-124554051640 4-171798691887' 2024-09-16T07:49:06.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:49:06.237 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T07:49:06.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=219043332132 2024-09-16T07:49:06.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 219043332132 2024-09-16T07:49:06.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803853 1-55834574914 2-81604378688 3-124554051640 4-171798691887 5-219043332132' 2024-09-16T07:49:06.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:49:06.392 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-16T07:49:06.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=257698037793 2024-09-16T07:49:06.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 257698037793 2024-09-16T07:49:06.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803853 1-55834574914 2-81604378688 3-124554051640 4-171798691887 5-219043332132 6-257698037793' 2024-09-16T07:49:06.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:49:06.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-16T07:49:06.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=304942678041 2024-09-16T07:49:06.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 304942678041 2024-09-16T07:49:06.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803853 1-55834574914 2-81604378688 3-124554051640 4-171798691887 5-219043332132 6-257698037793 7-304942678041' 2024-09-16T07:49:06.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:49:06.703 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-16T07:49:06.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=352187318289 2024-09-16T07:49:06.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 352187318289 2024-09-16T07:49:06.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803853 1-55834574914 2-81604378688 3-124554051640 4-171798691887 5-219043332132 6-257698037793 7-304942678041 8-352187318289' 2024-09-16T07:49:06.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:49:06.866 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803853 2024-09-16T07:49:06.866 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:49:06.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T07:49:06.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803853 2024-09-16T07:49:06.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:49:06.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803853 2024-09-16T07:49:06.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803853' 2024-09-16T07:49:06.872 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803853 2024-09-16T07:49:06.873 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:49:07.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803853 -lt 25769803853 2024-09-16T07:49:07.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:49:07.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-55834574914 2024-09-16T07:49:07.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:49:07.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T07:49:07.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-55834574914 2024-09-16T07:49:07.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:49:07.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=55834574914 2024-09-16T07:49:07.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 55834574914' 2024-09-16T07:49:07.330 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574914 2024-09-16T07:49:07.330 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T07:49:07.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 55834574914 -lt 55834574914 2024-09-16T07:49:07.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:49:07.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378688 2024-09-16T07:49:07.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:49:07.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T07:49:07.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:49:07.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378688 2024-09-16T07:49:07.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378688 2024-09-16T07:49:07.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378688' 2024-09-16T07:49:07.893 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 81604378688 2024-09-16T07:49:07.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T07:49:08.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378688 -lt 81604378688 2024-09-16T07:49:08.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:49:08.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-124554051640 2024-09-16T07:49:08.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:49:08.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T07:49:08.372 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:49:08.372 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-124554051640 2024-09-16T07:49:08.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=124554051640 2024-09-16T07:49:08.374 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 124554051640 2024-09-16T07:49:08.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 124554051640' 2024-09-16T07:49:08.374 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T07:49:08.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 124554051641 -lt 124554051640 2024-09-16T07:49:08.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:49:08.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-171798691887 2024-09-16T07:49:08.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:49:08.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T07:49:08.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-171798691887 2024-09-16T07:49:08.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:49:08.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=171798691887 2024-09-16T07:49:08.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 171798691887' 2024-09-16T07:49:08.824 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 171798691887 2024-09-16T07:49:08.824 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T07:49:09.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 171798691887 -lt 171798691887 2024-09-16T07:49:09.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:49:09.291 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-219043332132 2024-09-16T07:49:09.291 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:49:09.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T07:49:09.294 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-219043332132 2024-09-16T07:49:09.294 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:49:09.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=219043332132 2024-09-16T07:49:09.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 219043332132' 2024-09-16T07:49:09.296 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 219043332132 2024-09-16T07:49:09.296 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T07:49:09.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 219043332132 -lt 219043332132 2024-09-16T07:49:09.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:49:09.729 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 6-257698037793 2024-09-16T07:49:09.729 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:49:09.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=6 2024-09-16T07:49:09.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 6-257698037793 2024-09-16T07:49:09.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:49:09.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=257698037793 2024-09-16T07:49:09.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.6 seq 257698037793' 2024-09-16T07:49:09.735 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 257698037793 2024-09-16T07:49:09.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-16T07:49:10.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 257698037794 -lt 257698037793 2024-09-16T07:49:10.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:49:10.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 7-304942678041 2024-09-16T07:49:10.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:49:10.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=7 2024-09-16T07:49:10.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 7-304942678041 2024-09-16T07:49:10.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:49:10.227 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.7 seq 304942678041 2024-09-16T07:49:10.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=304942678041 2024-09-16T07:49:10.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.7 seq 304942678041' 2024-09-16T07:49:10.228 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-16T07:49:10.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 304942678041 -lt 304942678041 2024-09-16T07:49:10.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:49:10.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 8-352187318289 2024-09-16T07:49:10.698 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:49:10.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=8 2024-09-16T07:49:10.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 8-352187318289 2024-09-16T07:49:10.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:49:10.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=352187318289 2024-09-16T07:49:10.704 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.8 seq 352187318289 2024-09-16T07:49:10.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.8 seq 352187318289' 2024-09-16T07:49:10.704 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-16T07:49:11.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 352187318290 -lt 352187318289 2024-09-16T07:49:11.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T07:49:11.140 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:49:11.140 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:49:11.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T07:49:11.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T07:49:11.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T07:49:11.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T07:49:11.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T07:49:11.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T07:49:11.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T07:49:11.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T07:49:12.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T07:49:12.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T07:49:12.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:49:12.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:49:12.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T07:49:12.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T07:49:12.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T07:49:12.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:426: TEST_ec_recovery_multiple_errors: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-16T07:49:12.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-16T07:49:12.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-16T07:49:12.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-16T07:49:12.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:49:12.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-16T07:49:12.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:49:12.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-16T07:49:12.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:49:12.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-16T07:49:12.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:49:12.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-16T07:49:12.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-16T07:49:12.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:427: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-09-16T07:49:12.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=ec 2024-09-16T07:49:12.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T07:49:12.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T07:49:12.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T07:49:12.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-jerasure 2024-09-16T07:49:12.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T07:49:12.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=myobject 2024-09-16T07:49:12.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T07:49:12.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/test-erasure-eio 2024-09-16T07:49:12.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T07:49:12.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T07:49:12.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T07:49:12.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T07:49:12.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-jerasure myobject 2024-09-16T07:49:12.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T07:49:12.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-16T07:49:12.885 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-16T07:49:12.885 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:49:13.358 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T07:49:13.359 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:49:13.359 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:49:13.359 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T07:49:13.359 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-16T07:49:13.359 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:49:13.359 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:49:13.359 INFO:tasks.workunit.client.0.smithi162.stderr:7' 2024-09-16T07:49:13.360 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-09-16T07:49:13.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T07:49:13.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T07:49:13.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' ec '!=' ec ']' 2024-09-16T07:49:13.362 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/test-erasure-eio/3/type 2024-09-16T07:49:13.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T07:49:13.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T07:49:13.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T07:49:13.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T07:49:13.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T07:49:13.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T07:49:13.362 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T07:49:13.362 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T07:49:13.362 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T07:49:13.363 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T07:49:13.363 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:49:13.363 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:49:13.364 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:49:13.364 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.3.asok 2024-09-16T07:49:13.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79228/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T07:49:13.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T07:49:13.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T07:49:13.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T07:49:13.466 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T07:49:13.466 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T07:49:13.466 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T07:49:13.466 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:49:13.466 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:49:13.467 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:49:13.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.3.asok 2024-09-16T07:49:13.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T07:49:13.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-09-16T07:49:13.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:430: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 3 2024-09-16T07:49:13.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=ec 2024-09-16T07:49:13.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T07:49:13.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T07:49:13.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T07:49:13.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-jerasure 2024-09-16T07:49:13.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T07:49:13.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=myobject 2024-09-16T07:49:13.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T07:49:13.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/test-erasure-eio 2024-09-16T07:49:13.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T07:49:13.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=3 2024-09-16T07:49:13.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T07:49:13.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T07:49:13.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-jerasure myobject 2024-09-16T07:49:13.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T07:49:13.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-16T07:49:13.565 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-16T07:49:13.565 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:49:13.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T07:49:13.985 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:49:13.985 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:49:13.985 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T07:49:13.985 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-16T07:49:13.985 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:49:13.985 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:49:13.985 INFO:tasks.workunit.client.0.smithi162.stderr:7' 2024-09-16T07:49:13.985 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-09-16T07:49:13.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T07:49:13.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=0 2024-09-16T07:49:13.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' ec '!=' ec ']' 2024-09-16T07:49:13.986 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/test-erasure-eio/0/type 2024-09-16T07:49:13.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T07:49:13.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-16T07:49:13.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T07:49:13.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-09-16T07:49:13.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T07:49:13.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T07:49:13.989 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T07:49:13.989 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-09-16T07:49:13.990 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T07:49:13.990 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T07:49:13.991 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:49:13.991 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:49:13.991 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:49:13.991 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.0.asok 2024-09-16T07:49:13.992 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79228/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-16T07:49:14.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T07:49:14.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T07:49:14.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T07:49:14.087 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.0 2024-09-16T07:49:14.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T07:49:14.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T07:49:14.088 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:49:14.088 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:49:14.089 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:49:14.089 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.0.asok 2024-09-16T07:49:14.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T07:49:14.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.0.asok injectdataerr pool-jerasure myobject 3 2024-09-16T07:49:14.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:431: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 4 2024-09-16T07:49:14.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=ec 2024-09-16T07:49:14.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T07:49:14.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T07:49:14.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T07:49:14.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-jerasure 2024-09-16T07:49:14.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T07:49:14.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=myobject 2024-09-16T07:49:14.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T07:49:14.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/test-erasure-eio 2024-09-16T07:49:14.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T07:49:14.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=4 2024-09-16T07:49:14.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T07:49:14.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T07:49:14.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-jerasure myobject 2024-09-16T07:49:14.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T07:49:14.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-16T07:49:14.191 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-16T07:49:14.191 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:49:14.706 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T07:49:14.706 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:49:14.706 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:49:14.706 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T07:49:14.707 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-16T07:49:14.707 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:49:14.707 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:49:14.707 INFO:tasks.workunit.client.0.smithi162.stderr:7' 2024-09-16T07:49:14.707 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-09-16T07:49:14.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T07:49:14.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=6 2024-09-16T07:49:14.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' ec '!=' ec ']' 2024-09-16T07:49:14.708 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/test-erasure-eio/6/type 2024-09-16T07:49:14.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T07:49:14.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 6 bluestore_debug_inject_read_err true 2024-09-16T07:49:14.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T07:49:14.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=6 2024-09-16T07:49:14.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T07:49:14.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T07:49:14.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T07:49:14.711 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.6 2024-09-16T07:49:14.711 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.6 2024-09-16T07:49:14.711 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.6 ']' 2024-09-16T07:49:14.712 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:49:14.712 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:49:14.712 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:49:14.712 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.6.asok 2024-09-16T07:49:14.713 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79228/ceph-osd.6.asok config set bluestore_debug_inject_read_err true 2024-09-16T07:49:14.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T07:49:14.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T07:49:14.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T07:49:14.819 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.6 2024-09-16T07:49:14.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.6 2024-09-16T07:49:14.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.6 ']' 2024-09-16T07:49:14.821 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:49:14.821 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:49:14.822 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:49:14.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.6.asok 2024-09-16T07:49:14.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T07:49:14.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.6.asok injectdataerr pool-jerasure myobject 4 2024-09-16T07:49:14.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T07:49:14.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: get_osds pool-jerasure myobject 2024-09-16T07:49:14.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T07:49:14.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-16T07:49:14.924 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-16T07:49:14.924 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:49:15.440 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T07:49:15.440 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:49:15.440 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:49:15.441 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T07:49:15.441 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-16T07:49:15.441 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:49:15.441 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:49:15.441 INFO:tasks.workunit.client.0.smithi162.stderr:7' 2024-09-16T07:49:15.441 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-09-16T07:49:15.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: local -a initial_osds 2024-09-16T07:49:15.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:434: TEST_ec_recovery_multiple_errors: local last_osd=7 2024-09-16T07:49:15.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:436: TEST_ec_recovery_multiple_errors: kill_daemons td/test-erasure-eio TERM osd.7 2024-09-16T07:49:15.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:49:15.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:49:15.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:49:15.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:49:15.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:49:16.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:49:16.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:437: TEST_ec_recovery_multiple_errors: ceph osd down 7 2024-09-16T07:49:17.627 INFO:tasks.workunit.client.0.smithi162.stderr:osd.7 is already down. 2024-09-16T07:49:17.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:438: TEST_ec_recovery_multiple_errors: ceph osd out 7 2024-09-16T07:49:18.372 INFO:tasks.workunit.client.0.smithi162.stderr:osd.7 is already out. 2024-09-16T07:49:18.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:441: TEST_ec_recovery_multiple_errors: wait_for_clean 2024-09-16T07:49:18.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T07:49:18.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T07:49:18.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T07:49:18.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T07:49:18.382 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T07:49:18.383 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T07:49:18.383 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T07:49:18.383 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T07:49:18.384 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T07:49:18.384 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T07:49:18.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T07:49:18.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T07:49:18.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T07:49:18.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T07:49:18.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T07:49:19.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T07:49:19.066 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:49:19.066 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:49:19.066 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:49:19.066 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:49:19.066 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:49:19.066 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-16T07:49:19.066 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-16T07:49:19.066 INFO:tasks.workunit.client.0.smithi162.stderr:8' 2024-09-16T07:49:19.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T07:49:19.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:49:19.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T07:49:19.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803858 2024-09-16T07:49:19.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803858 2024-09-16T07:49:19.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803858' 2024-09-16T07:49:19.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:49:19.339 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T07:49:19.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=55834574919 2024-09-16T07:49:19.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 55834574919 2024-09-16T07:49:19.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803858 1-55834574919' 2024-09-16T07:49:19.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:49:19.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T07:49:19.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378693 2024-09-16T07:49:19.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378693 2024-09-16T07:49:19.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803858 1-55834574919 2-81604378693' 2024-09-16T07:49:19.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:49:19.919 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T07:49:20.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=124554051645 2024-09-16T07:49:20.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 124554051645 2024-09-16T07:49:20.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803858 1-55834574919 2-81604378693 3-124554051645' 2024-09-16T07:49:20.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:49:20.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T07:49:20.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=171798691891 2024-09-16T07:49:20.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 171798691891 2024-09-16T07:49:20.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803858 1-55834574919 2-81604378693 3-124554051645 4-171798691891' 2024-09-16T07:49:20.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:49:20.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T07:49:20.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=219043332137 2024-09-16T07:49:20.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 219043332137 2024-09-16T07:49:20.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803858 1-55834574919 2-81604378693 3-124554051645 4-171798691891 5-219043332137' 2024-09-16T07:49:20.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:49:20.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-16T07:49:20.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=257698037798 2024-09-16T07:49:20.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 257698037798 2024-09-16T07:49:20.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803858 1-55834574919 2-81604378693 3-124554051645 4-171798691891 5-219043332137 6-257698037798' 2024-09-16T07:49:20.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:49:20.677 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-16T07:49:20.791 INFO:tasks.workunit.client.0.smithi162.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T07:49:20.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq= 2024-09-16T07:49:20.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z '' 2024-09-16T07:49:20.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: flush_pg_stats: continue 2024-09-16T07:49:20.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:49:20.794 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-16T07:49:20.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=352187318294 2024-09-16T07:49:20.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 352187318294 2024-09-16T07:49:20.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803858 1-55834574919 2-81604378693 3-124554051645 4-171798691891 5-219043332137 6-257698037798 8-352187318294' 2024-09-16T07:49:20.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:49:20.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803858 2024-09-16T07:49:20.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:49:20.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T07:49:20.950 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803858 2024-09-16T07:49:20.950 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:49:20.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803858 2024-09-16T07:49:20.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803858' 2024-09-16T07:49:20.953 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803858 2024-09-16T07:49:20.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:49:21.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803858 -lt 25769803858 2024-09-16T07:49:21.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:49:21.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-55834574919 2024-09-16T07:49:21.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:49:21.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T07:49:21.425 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-55834574919 2024-09-16T07:49:21.425 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:49:21.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=55834574919 2024-09-16T07:49:21.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 55834574919' 2024-09-16T07:49:21.428 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574919 2024-09-16T07:49:21.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T07:49:21.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 55834574919 -lt 55834574919 2024-09-16T07:49:21.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:49:21.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378693 2024-09-16T07:49:21.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:49:21.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T07:49:21.844 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378693 2024-09-16T07:49:21.844 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:49:21.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378693 2024-09-16T07:49:21.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378693' 2024-09-16T07:49:21.846 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 81604378693 2024-09-16T07:49:21.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T07:49:22.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378693 -lt 81604378693 2024-09-16T07:49:22.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:49:22.296 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-124554051645 2024-09-16T07:49:22.296 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:49:22.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T07:49:22.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-124554051645 2024-09-16T07:49:22.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:49:22.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=124554051645 2024-09-16T07:49:22.301 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 124554051645 2024-09-16T07:49:22.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 124554051645' 2024-09-16T07:49:22.302 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T07:49:22.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 124554051646 -lt 124554051645 2024-09-16T07:49:22.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:49:22.803 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-171798691891 2024-09-16T07:49:22.804 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:49:22.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T07:49:22.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-171798691891 2024-09-16T07:49:22.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:49:22.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=171798691891 2024-09-16T07:49:22.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 171798691891' 2024-09-16T07:49:22.811 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 171798691891 2024-09-16T07:49:22.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T07:49:23.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 171798691892 -lt 171798691891 2024-09-16T07:49:23.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:49:23.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-219043332137 2024-09-16T07:49:23.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:49:23.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T07:49:23.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:49:23.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-219043332137 2024-09-16T07:49:23.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=219043332137 2024-09-16T07:49:23.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 219043332137' 2024-09-16T07:49:23.249 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 219043332137 2024-09-16T07:49:23.250 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T07:49:23.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 219043332137 -lt 219043332137 2024-09-16T07:49:23.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:49:23.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 6-257698037798 2024-09-16T07:49:23.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:49:23.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=6 2024-09-16T07:49:23.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 6-257698037798 2024-09-16T07:49:23.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:49:23.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=257698037798 2024-09-16T07:49:23.658 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 257698037798 2024-09-16T07:49:23.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.6 seq 257698037798' 2024-09-16T07:49:23.658 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-16T07:49:24.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 257698037799 -lt 257698037798 2024-09-16T07:49:24.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:49:24.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 8-352187318294 2024-09-16T07:49:24.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:49:24.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=8 2024-09-16T07:49:24.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 8-352187318294 2024-09-16T07:49:24.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:49:24.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=352187318294 2024-09-16T07:49:24.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.8 seq 352187318294' 2024-09-16T07:49:24.075 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.8 seq 352187318294 2024-09-16T07:49:24.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-16T07:49:24.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 352187318294 -lt 352187318294 2024-09-16T07:49:24.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T07:49:24.495 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:49:24.495 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:49:25.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T07:49:25.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T07:49:25.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T07:49:25.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T07:49:25.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T07:49:25.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T07:49:25.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T07:49:25.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T07:49:25.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T07:49:25.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T07:49:25.497 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:49:25.497 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:49:26.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T07:49:26.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T07:49:26.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T07:49:26.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:443: TEST_ec_recovery_multiple_errors: rados_get td/test-erasure-eio pool-jerasure myobject 2024-09-16T07:49:26.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T07:49:26.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T07:49:26.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-09-16T07:49:26.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-16T07:49:26.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-16T07:49:26.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-09-16T07:49:26.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-16T07:49:26.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-16T07:49:26.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:445: TEST_ec_recovery_multiple_errors: delete_erasure_coded_pool pool-jerasure 2024-09-16T07:49:26.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T07:49:26.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-16T07:49:26.782 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-16T07:49:26.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-16T07:49:27.958 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-16T07:49:27.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-16T07:49:27.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T07:49:27.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:49:27.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T07:49:27.968 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:49:27.968 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:49:27.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:49:27.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:49:27.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:49:28.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:49:28.293 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:49:28.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:49:28.296 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:49:28.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:49:28.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:49:28.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:49:28.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:49:28.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:49:28.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:49:28.302 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:49:28.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:49:28.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:49:28.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T07:49:28.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:49:28.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:49:28.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:49:28.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T07:49:28.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:49:28.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:49:28.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-16T07:49:28.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-16T07:49:28.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-16T07:49:28.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-16T07:49:28.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T07:49:28.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:49:28.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T07:49:28.358 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:49:28.358 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:49:28.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:49:28.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:49:28.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:49:28.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:49:28.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:49:28.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:49:28.366 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:49:28.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:49:28.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:49:28.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:49:28.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:49:28.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:49:28.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:49:28.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:49:28.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:49:28.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:49:28.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T07:49:28.376 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:49:28.376 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:49:28.376 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:49:28.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T07:49:28.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:49:28.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:49:28.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-16T07:49:28.380 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T07:49:28.380 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:49:28.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:49:28.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79228 2024-09-16T07:49:28.383 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T07:49:28.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T07:49:28.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T07:49:28.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-16T07:49:28.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-16T07:49:28.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-16T07:49:28.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T07:49:28.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T07:49:28.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T07:49:28.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-16T07:49:28.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-16T07:49:28.848 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T07:49:28.848 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:49:28.848 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:49:28.849 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:49:28.849 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:49:28.849 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:49:28.849 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:49:28.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T07:49:29.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T07:49:29.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T07:49:29.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:49:29.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:49:29.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T07:49:29.071 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:49:29.071 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T07:49:29.071 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:49:29.071 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:49:29.072 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:49:29.072 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:49:29.072 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:49:29.072 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T07:49:29.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:49:29.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get fsid 2024-09-16T07:49:29.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T07:49:29.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:49:29.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:49:29.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T07:49:29.171 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T07:49:29.171 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:49:29.171 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:49:29.171 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:49:29.172 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:49:29.172 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:49:29.172 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:49:29.172 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T07:49:29.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:49:29.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get mon_host 2024-09-16T07:49:29.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-16T07:49:29.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-16T07:49:29.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T07:49:29.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T07:49:29.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T07:49:29.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-16T07:49:29.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T07:49:30.183 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T07:49:30.183 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:49:30.183 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:49:30.184 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:49:30.184 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:49:30.184 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:49:30.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:49:30.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:49:30.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:49:30.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-16T07:49:30.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T07:49:30.577 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-16T07:49:30.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T07:49:31.588 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-16T07:49:31.589 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:49:31.589 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:49:31.589 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:49:31.589 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:49:31.589 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:49:31.590 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T07:49:31.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-16T07:49:31.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-mon.a.asok log flush 2024-09-16T07:49:31.677 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:49:31.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-16T07:49:31.689 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:49:28.899+0000 7f801f537d40 0 load: jerasure load: lrc 2024-09-16T07:49:31.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects td/test-erasure-eio 2024-09-16T07:49:31.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:451: TEST_ec_recovery_multiple_objects: local dir=td/test-erasure-eio 2024-09-16T07:49:31.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:452: TEST_ec_recovery_multiple_objects: local objname=myobject 2024-09-16T07:49:31.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:454: TEST_ec_recovery_multiple_objects: ORIG_ARGS='--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:49:31.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:455: TEST_ec_recovery_multiple_objects: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-16T07:49:31.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:456: TEST_ec_recovery_multiple_objects: setup_osds 7 2024-09-16T07:49:31.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-09-16T07:49:31.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-16T07:49:31.691 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-09-16T07:49:31.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-09-16T07:49:31.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:49:31.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-16T07:49:31.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:49:31.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:49:31.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T07:49:31.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:49:31.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-16T07:49:31.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-16T07:49:31.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:49:31.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:49:31.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:49:31.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-16T07:49:31.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-16T07:49:31.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:49:31.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:49:31.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:49:31.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:49:31.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:49:31.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:49:31.698 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:49:31.698 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:49:31.698 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:49:31.698 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:49:31.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:49:31.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:49:31.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:49:31.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:49:31.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:49:31.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:49:31.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:49:31.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:49:31.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:49:31.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:49:31.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:49:31.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:49:31.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-16T07:49:31.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:49:31.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=529bfd98-934d-455a-bad9-f4626ffce003 2024-09-16T07:49:31.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 529bfd98-934d-455a-bad9-f4626ffce003' 2024-09-16T07:49:31.701 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 529bfd98-934d-455a-bad9-f4626ffce003 2024-09-16T07:49:31.702 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:49:31.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAL4+dmqxGEKxAAA4OWs27DCt1s0k98TCQpQQ== 2024-09-16T07:49:31.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAL4+dmqxGEKxAAA4OWs27DCt1s0k98TCQpQQ=="}' 2024-09-16T07:49:31.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 529bfd98-934d-455a-bad9-f4626ffce003 -i td/test-erasure-eio/0/new.json 2024-09-16T07:49:32.054 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:49:32.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-16T07:49:32.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAL4+dmqxGEKxAAA4OWs27DCt1s0k98TCQpQQ== --osd-uuid 529bfd98-934d-455a-bad9-f4626ffce003 2024-09-16T07:49:32.109 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:49:32.103+0000 7f39ad803880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:49:32.112 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:49:32.107+0000 7f39ad803880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:49:32.115 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:49:32.111+0000 7f39ad803880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:49:32.115 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:49:32.111+0000 7f39ad803880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T07:49:32.116 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:49:32.111+0000 7f39ad803880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T07:49:32.116 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:49:32.111+0000 7f39ad803880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T07:49:32.156 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:49:32.151+0000 7f39ad803880 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-16T07:49:37.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-16T07:49:37.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:49:37.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T07:49:37.878 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T07:49:37.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:49:38.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T07:49:38.436 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T07:49:38.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:49:38.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:49:38.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:49:38.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:49:38.471 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:49:38.466+0000 7f021ec89880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:49:38.490 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:49:38.482+0000 7f021ec89880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:49:38.492 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:49:38.486+0000 7f021ec89880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:49:38.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T07:49:38.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:49:38.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T07:49:38.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:49:38.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:49:38.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:49:38.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:49:38.865 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:49:38.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:49:38.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:49:39.041 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:49:39.034+0000 7f021ec89880 -1 Falling back to public interface 2024-09-16T07:49:39.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:49:40.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:49:40.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:49:40.328 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:49:40.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:49:40.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:49:40.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:49:40.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:49:41.686 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:49:41.682+0000 7f021ec89880 -1 osd.0 0 log_to_monitors true 2024-09-16T07:49:41.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:49:41.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:49:41.756 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:49:41.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:49:41.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:49:41.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:49:42.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:49:43.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:49:43.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:49:43.315 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:49:43.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:49:43.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:49:43.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:49:43.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:49:44.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:49:44.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:49:44.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:49:44.841 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:49:44.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:49:44.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:49:45.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:49:46.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:49:46.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:49:46.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:49:46.278 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:49:46.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:49:46.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:49:46.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:49:47.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:49:47.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:49:47.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:49:47.818 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:49:47.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:49:47.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:49:48.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:49:49.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:49:49.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:49:49.281 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:49:49.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:49:49.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:49:49.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:49:49.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:49:50.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:49:50.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:49:50.818 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:49:50.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:49:50.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:49:50.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:49:51.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:49:52.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:49:52.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:49:52.269 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:49:52.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:49:52.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:49:52.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:49:52.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:49:53.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:49:53.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:49:53.815 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:49:53.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:49:53.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:49:53.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:49:54.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:49:55.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:49:55.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:49:55.350 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:49:55.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:49:55.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:49:55.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:49:56.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:49:57.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:49:57.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:49:57.385 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:49:57.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:49:57.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:49:57.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:49:58.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:49:59.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:49:59.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:49:59.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:49:59.355 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:49:59.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:49:59.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:50:00.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:01.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:01.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:01.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:50:01.179 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:50:01.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:01.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:50:02.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:03.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:03.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:03.434 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:50:03.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:50:03.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:03.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:50:04.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:05.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:05.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:05.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:50:05.305 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:50:05.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:05.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:50:05.813 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/653984593,v1:127.0.0.1:6803/653984593] [v2:127.0.0.1:6804/653984593,v1:127.0.0.1:6805/653984593] exists,up 529bfd98-934d-455a-bad9-f4626ffce003 2024-09-16T07:50:05.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:50:05.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:50:05.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:50:05.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:50:05.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-16T07:50:05.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:50:05.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:50:05.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T07:50:05.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:50:05.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-16T07:50:05.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-16T07:50:05.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:50:05.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:50:05.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:50:05.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-16T07:50:05.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-16T07:50:05.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:50:05.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:50:05.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:50:05.824 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:50:05.824 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:50:05.824 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:50:05.825 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:50:05.825 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:50:05.825 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:50:05.825 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:50:05.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:50:05.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:50:05.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:50:05.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:50:05.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:50:05.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:50:05.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:50:05.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:50:05.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:50:05.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:50:05.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:50:05.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:50:05.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-16T07:50:05.827 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:50:05.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=55ed3e6e-98d1-4e8a-9833-b4ab1d2e746d 2024-09-16T07:50:05.828 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 55ed3e6e-98d1-4e8a-9833-b4ab1d2e746d 2024-09-16T07:50:05.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 55ed3e6e-98d1-4e8a-9833-b4ab1d2e746d' 2024-09-16T07:50:05.829 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:50:05.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAt4+dmInVTMhAALz266tmLkFoXSte4XNTQUw== 2024-09-16T07:50:05.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAt4+dmInVTMhAALz266tmLkFoXSte4XNTQUw=="}' 2024-09-16T07:50:05.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 55ed3e6e-98d1-4e8a-9833-b4ab1d2e746d -i td/test-erasure-eio/1/new.json 2024-09-16T07:50:06.660 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:50:06.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-16T07:50:06.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAt4+dmInVTMhAALz266tmLkFoXSte4XNTQUw== --osd-uuid 55ed3e6e-98d1-4e8a-9833-b4ab1d2e746d 2024-09-16T07:50:06.723 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:50:06.718+0000 7fe4a07d1880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:50:06.727 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:50:06.726+0000 7fe4a07d1880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:50:06.729 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:50:06.726+0000 7fe4a07d1880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:50:06.729 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:50:06.726+0000 7fe4a07d1880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T07:50:06.730 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:50:06.726+0000 7fe4a07d1880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T07:50:06.730 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:50:06.726+0000 7fe4a07d1880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T07:50:06.802 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:50:06.798+0000 7fe4a07d1880 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-16T07:50:13.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-16T07:50:13.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:50:13.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T07:50:13.461 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-16T07:50:13.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:50:14.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T07:50:14.059 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-16T07:50:14.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:50:14.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:50:14.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:50:14.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:50:14.097 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:50:14.094+0000 7f82fc6c7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:50:14.100 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:50:14.098+0000 7f82fc6c7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:50:14.102 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:50:14.098+0000 7f82fc6c7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:50:14.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T07:50:14.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:50:14.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T07:50:14.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:50:14.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:50:14.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:14.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:50:14.523 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:50:14.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:14.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:50:14.676 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:50:14.674+0000 7f82fc6c7880 -1 Falling back to public interface 2024-09-16T07:50:14.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:15.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:15.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:15.944 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:50:15.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:50:15.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:15.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:50:16.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:17.314 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:50:17.310+0000 7f82fc6c7880 -1 osd.1 0 log_to_monitors true 2024-09-16T07:50:17.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:17.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:17.373 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:50:17.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:50:17.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:17.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:50:18.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:19.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:19.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:19.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:50:19.062 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:50:19.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:19.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:50:19.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:20.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:20.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:20.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:50:20.581 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:50:20.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:20.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:50:21.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:22.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:22.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:22.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:50:22.107 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:50:22.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:22.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:50:22.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:23.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:23.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:23.778 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:50:23.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:50:23.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:23.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:50:24.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:25.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:25.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:25.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:50:25.362 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:50:25.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:25.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:50:25.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:26.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:26.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:26.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:50:26.839 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:50:26.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:26.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:50:27.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:28.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:28.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:28.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:50:28.295 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:50:28.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:28.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:50:28.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:29.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:29.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:29.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:50:29.774 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:50:29.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:29.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:50:30.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:31.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:31.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:31.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:50:31.352 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:50:31.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:31.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:50:32.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:33.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:33.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:33.374 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:50:33.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:50:33.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:33.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:50:34.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:35.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:35.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:35.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:50:35.549 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:50:35.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:35.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:50:36.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:37.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:37.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:37.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:50:37.155 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:50:37.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:37.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:50:37.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:38.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:38.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:38.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:50:38.837 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:50:38.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:38.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:50:39.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:40.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:40.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:40.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:50:40.744 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:50:40.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:40.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:50:41.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:42.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:42.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:42.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:50:42.244 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:50:42.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:42.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:50:43.015 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1844251732,v1:127.0.0.1:6811/1844251732] [v2:127.0.0.1:6812/1844251732,v1:127.0.0.1:6813/1844251732] exists,up 55ed3e6e-98d1-4e8a-9833-b4ab1d2e746d 2024-09-16T07:50:43.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:50:43.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:50:43.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:50:43.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:50:43.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-16T07:50:43.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:50:43.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:50:43.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T07:50:43.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:50:43.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-16T07:50:43.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-16T07:50:43.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:50:43.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:50:43.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:50:43.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-16T07:50:43.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-16T07:50:43.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:50:43.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:50:43.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:50:43.020 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:50:43.020 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:50:43.020 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:50:43.020 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:50:43.020 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:50:43.020 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:50:43.022 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:50:43.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:50:43.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:50:43.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:50:43.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:50:43.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:50:43.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:50:43.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:50:43.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:50:43.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:50:43.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:50:43.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:50:43.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:50:43.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-16T07:50:43.023 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:50:43.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4d30814a-96be-4c43-b88f-ae0ad3b819c9 2024-09-16T07:50:43.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4d30814a-96be-4c43-b88f-ae0ad3b819c9' 2024-09-16T07:50:43.024 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 4d30814a-96be-4c43-b88f-ae0ad3b819c9 2024-09-16T07:50:43.024 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:50:43.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBT4+dmVGzgAhAAhxjle9CHWk0WjHZ+p7YqrA== 2024-09-16T07:50:43.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBT4+dmVGzgAhAAhxjle9CHWk0WjHZ+p7YqrA=="}' 2024-09-16T07:50:43.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4d30814a-96be-4c43-b88f-ae0ad3b819c9 -i td/test-erasure-eio/2/new.json 2024-09-16T07:50:43.638 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:50:43.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-16T07:50:43.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBT4+dmVGzgAhAAhxjle9CHWk0WjHZ+p7YqrA== --osd-uuid 4d30814a-96be-4c43-b88f-ae0ad3b819c9 2024-09-16T07:50:43.698 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:50:43.693+0000 7f49028cd880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:50:43.728 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:50:43.725+0000 7f49028cd880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:50:43.730 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:50:43.729+0000 7f49028cd880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:50:43.731 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:50:43.729+0000 7f49028cd880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T07:50:43.731 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:50:43.729+0000 7f49028cd880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T07:50:43.731 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:50:43.729+0000 7f49028cd880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T07:50:43.819 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:50:43.817+0000 7f49028cd880 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-16T07:50:49.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-16T07:50:49.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:50:49.930 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-16T07:50:49.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T07:50:49.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:50:50.589 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-16T07:50:50.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T07:50:50.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:50:50.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:50:50.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:50:50.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:50:50.628 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:50:50.625+0000 7f2f2c285880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:50:50.664 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:50:50.661+0000 7f2f2c285880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:50:50.666 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:50:50.661+0000 7f2f2c285880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:50:51.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T07:50:51.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:50:51.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T07:50:51.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:50:51.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:50:51.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:51.057 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:50:51.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:50:51.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:51.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:50:51.241 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:50:51.237+0000 7f2f2c285880 -1 Falling back to public interface 2024-09-16T07:50:51.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:52.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:52.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:52.493 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:50:52.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:50:52.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:52.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:50:52.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:53.722 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:50:53.721+0000 7f2f2c285880 -1 osd.2 0 log_to_monitors true 2024-09-16T07:50:53.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:53.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:53.910 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:50:53.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:50:53.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:53.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:50:54.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:55.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:55.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:55.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:50:55.431 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:50:55.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:55.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:50:55.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:56.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:56.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:56.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:50:56.889 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:50:56.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:56.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:50:57.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:58.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:58.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:58.319 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:50:58.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:50:58.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:58.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:50:58.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:50:59.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:50:59.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:50:59.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:50:59.935 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:50:59.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:50:59.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:51:00.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:01.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:01.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:01.564 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:51:01.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:51:01.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:01.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:51:02.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:03.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:03.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:03.033 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:51:03.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:51:03.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:03.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:51:03.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:04.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:04.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:04.540 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:51:04.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:51:04.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:04.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:51:04.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:05.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:05.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:05.985 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:51:05.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:51:05.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:05.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:51:06.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:07.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:07.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:07.505 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:51:07.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:51:07.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:07.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:51:08.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:09.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:09.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:09.352 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:51:09.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:51:09.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:09.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:51:10.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:11.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:11.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:11.625 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:51:11.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:51:11.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:11.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:51:12.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:13.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:13.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:13.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:51:13.471 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:51:13.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:13.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:51:14.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:15.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:15.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:15.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:51:15.516 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:51:15.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:15.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:51:16.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:17.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:17.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:17.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:51:17.391 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:51:17.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:17.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:51:18.155 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2070483636,v1:127.0.0.1:6819/2070483636] [v2:127.0.0.1:6820/2070483636,v1:127.0.0.1:6821/2070483636] exists,up 4d30814a-96be-4c43-b88f-ae0ad3b819c9 2024-09-16T07:51:18.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:51:18.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:51:18.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:51:18.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:51:18.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-16T07:51:18.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:51:18.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:51:18.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T07:51:18.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:51:18.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-16T07:51:18.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-16T07:51:18.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:51:18.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:51:18.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:51:18.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-16T07:51:18.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-16T07:51:18.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:51:18.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:51:18.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:51:18.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:51:18.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:51:18.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:51:18.159 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:51:18.160 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:51:18.160 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:51:18.161 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:51:18.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:51:18.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:51:18.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:51:18.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:51:18.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:51:18.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:51:18.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:51:18.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:51:18.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:51:18.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:51:18.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:51:18.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:51:18.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-16T07:51:18.162 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:51:18.163 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 b588fe42-689c-47ad-bc58-4ba3fe10cb2a 2024-09-16T07:51:18.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b588fe42-689c-47ad-bc58-4ba3fe10cb2a 2024-09-16T07:51:18.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 b588fe42-689c-47ad-bc58-4ba3fe10cb2a' 2024-09-16T07:51:18.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:51:18.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB24+dmPc9zCxAAfZHwQTzm+LkwBDfllLON2A== 2024-09-16T07:51:18.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB24+dmPc9zCxAAfZHwQTzm+LkwBDfllLON2A=="}' 2024-09-16T07:51:18.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b588fe42-689c-47ad-bc58-4ba3fe10cb2a -i td/test-erasure-eio/3/new.json 2024-09-16T07:51:19.279 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:51:19.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-16T07:51:19.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB24+dmPc9zCxAAfZHwQTzm+LkwBDfllLON2A== --osd-uuid b588fe42-689c-47ad-bc58-4ba3fe10cb2a 2024-09-16T07:51:19.342 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:51:19.336+0000 7fdfc0163880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:51:19.346 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:51:19.340+0000 7fdfc0163880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:51:19.348 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:51:19.344+0000 7fdfc0163880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:51:19.348 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:51:19.344+0000 7fdfc0163880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T07:51:19.348 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:51:19.344+0000 7fdfc0163880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T07:51:19.348 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:51:19.344+0000 7fdfc0163880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T07:51:19.412 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:51:19.408+0000 7fdfc0163880 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-16T07:51:26.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-16T07:51:26.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:51:26.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T07:51:26.014 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-16T07:51:26.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:51:26.643 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-16T07:51:26.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T07:51:26.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:51:26.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:51:26.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:51:26.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:51:26.681 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:51:26.676+0000 7fa372c55880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:51:26.710 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:51:26.708+0000 7fa372c55880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:51:26.712 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:51:26.708+0000 7fa372c55880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:51:27.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T07:51:27.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:51:27.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T07:51:27.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:51:27.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:51:27.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:27.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:51:27.085 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:51:27.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:27.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:51:27.260 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:51:27.256+0000 7fa372c55880 -1 Falling back to public interface 2024-09-16T07:51:27.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:28.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:28.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:28.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:51:28.515 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:51:28.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:28.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:51:29.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:29.988 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:51:29.984+0000 7fa372c55880 -1 osd.3 0 log_to_monitors true 2024-09-16T07:51:30.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:30.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:30.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:51:30.030 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:51:30.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:30.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:51:30.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:31.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:31.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:31.515 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:51:31.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:51:31.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:31.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:51:32.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:33.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:33.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:33.170 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:51:33.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:51:33.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:33.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:51:33.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:34.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:34.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:34.636 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:51:34.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:51:34.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:34.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:51:35.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:36.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:36.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:36.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:51:36.058 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:51:36.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:36.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:51:36.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:37.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:37.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:37.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:51:37.577 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:51:37.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:37.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:51:38.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:39.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:39.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:39.208 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:51:39.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:51:39.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:39.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:51:39.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:40.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:40.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:40.644 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:51:40.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:51:40.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:40.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:51:41.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:42.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:42.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:42.143 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:51:42.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:51:42.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:42.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:51:42.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:43.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:43.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:43.609 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:51:43.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:51:43.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:43.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:51:44.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:45.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:45.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:45.287 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:51:45.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:51:45.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:45.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:51:45.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:46.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:46.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:46.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:51:46.866 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:51:46.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:46.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:51:47.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:48.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:48.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:48.918 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:51:48.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:51:48.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:48.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:51:49.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:50.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:50.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:50.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:51:50.665 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:51:50.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:50.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:51:51.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:52.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:52.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:52.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:51:52.106 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:51:52.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:52.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:51:53.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:54.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:54.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:54.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:51:54.037 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:51:54.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:54.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:51:54.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:51:55.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:51:55.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:51:55.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T07:51:55.800 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-16T07:51:55.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:51:55.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:51:56.245 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4045920294,v1:127.0.0.1:6827/4045920294] [v2:127.0.0.1:6828/4045920294,v1:127.0.0.1:6829/4045920294] exists,up b588fe42-689c-47ad-bc58-4ba3fe10cb2a 2024-09-16T07:51:56.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:51:56.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:51:56.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:51:56.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:51:56.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-16T07:51:56.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:51:56.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:51:56.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-16T07:51:56.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:51:56.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-16T07:51:56.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-16T07:51:56.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:51:56.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:51:56.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:51:56.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-16T07:51:56.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-16T07:51:56.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:51:56.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:51:56.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:51:56.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:51:56.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:51:56.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:51:56.249 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:51:56.249 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:51:56.249 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:51:56.251 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:51:56.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:51:56.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:51:56.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:51:56.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:51:56.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:51:56.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:51:56.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:51:56.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:51:56.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:51:56.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:51:56.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:51:56.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:51:56.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-16T07:51:56.252 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:51:56.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=889b03cd-30b5-46ed-a626-c13892cfddf9 2024-09-16T07:51:56.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 889b03cd-30b5-46ed-a626-c13892cfddf9' 2024-09-16T07:51:56.254 INFO:tasks.workunit.client.0.smithi162.stdout:add osd4 889b03cd-30b5-46ed-a626-c13892cfddf9 2024-09-16T07:51:56.254 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:51:56.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCc4+dmYKDpEBAA5cZcm5zIrL/SQH+atFVpzQ== 2024-09-16T07:51:56.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCc4+dmYKDpEBAA5cZcm5zIrL/SQH+atFVpzQ=="}' 2024-09-16T07:51:56.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 889b03cd-30b5-46ed-a626-c13892cfddf9 -i td/test-erasure-eio/4/new.json 2024-09-16T07:51:56.943 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:51:56.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-16T07:51:56.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCc4+dmYKDpEBAA5cZcm5zIrL/SQH+atFVpzQ== --osd-uuid 889b03cd-30b5-46ed-a626-c13892cfddf9 2024-09-16T07:51:56.996 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:51:56.992+0000 7f9bfc5b0880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:51:57.175 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:51:57.172+0000 7f9bfc5b0880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:51:57.177 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:51:57.172+0000 7f9bfc5b0880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:51:57.178 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:51:57.176+0000 7f9bfc5b0880 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-16T07:51:57.179 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:51:57.176+0000 7f9bfc5b0880 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-16T07:51:57.179 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:51:57.176+0000 7f9bfc5b0880 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-16T07:51:57.298 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:51:57.292+0000 7f9bfc5b0880 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-16T07:52:03.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-16T07:52:03.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:52:03.901 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd4 key to auth repository 2024-09-16T07:52:03.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-16T07:52:03.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:52:04.465 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.4 2024-09-16T07:52:04.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-16T07:52:04.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:52:04.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:52:04.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:52:04.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:52:04.505 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:52:04.499+0000 7f9e47757880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:52:04.533 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:52:04.527+0000 7f9e47757880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:52:04.535 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:52:04.531+0000 7f9e47757880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:52:04.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-16T07:52:04.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:52:04.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-16T07:52:04.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:52:04.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:52:04.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:04.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:52:04.937 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:52:04.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:04.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:52:05.089 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:52:05.083+0000 7f9e47757880 -1 Falling back to public interface 2024-09-16T07:52:05.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:06.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:06.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:06.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:52:06.441 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:52:06.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:06.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:52:06.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:07.579 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:52:07.575+0000 7f9e47757880 -1 osd.4 0 log_to_monitors true 2024-09-16T07:52:07.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:07.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:07.928 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:52:07.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:52:07.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:07.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:52:08.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:09.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:09.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:09.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:52:09.435 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:52:09.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:09.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:52:10.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:11.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:11.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:11.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:52:11.353 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:52:11.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:11.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:52:11.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:12.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:12.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:12.895 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:52:12.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:52:12.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:12.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:52:13.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:14.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:14.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:14.420 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:52:14.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:52:14.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:14.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:52:14.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:15.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:15.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:15.901 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:52:15.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:52:15.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:15.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:52:16.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:17.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:17.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:17.422 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:52:17.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:52:17.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:17.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:52:17.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:18.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:18.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:18.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:52:18.916 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:52:18.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:18.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:52:19.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:20.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:20.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:20.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:52:20.406 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:52:20.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:20.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:52:20.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:21.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:21.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:21.917 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:52:21.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:52:21.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:21.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:52:23.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:24.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:24.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:24.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:52:24.136 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:52:24.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:24.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:52:25.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:26.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:26.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:26.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:52:26.092 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:52:26.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:26.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:52:26.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:27.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:27.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:27.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:52:27.978 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:52:27.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:27.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:52:28.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:29.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:29.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:29.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:52:29.952 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:52:29.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:29.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:52:30.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:31.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:31.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:31.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:52:31.876 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:52:31.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:31.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:52:32.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:33.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:33.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:33.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:52:33.855 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:52:33.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:33.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:52:34.429 INFO:tasks.workunit.client.0.smithi162.stdout:osd.4 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3896638226,v1:127.0.0.1:6835/3896638226] [v2:127.0.0.1:6836/3896638226,v1:127.0.0.1:6837/3896638226] exists,up 889b03cd-30b5-46ed-a626-c13892cfddf9 2024-09-16T07:52:34.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:52:34.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:52:34.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:52:34.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:52:34.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-16T07:52:34.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:52:34.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:52:34.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-16T07:52:34.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:52:34.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-16T07:52:34.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-16T07:52:34.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:52:34.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:52:34.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:52:34.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-16T07:52:34.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-16T07:52:34.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:52:34.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:52:34.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:52:34.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:52:34.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:52:34.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:52:34.433 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:52:34.433 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:52:34.433 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:52:34.434 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:52:34.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:52:34.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:52:34.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:52:34.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:52:34.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:52:34.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:52:34.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:52:34.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:52:34.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:52:34.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:52:34.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:52:34.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:52:34.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-16T07:52:34.436 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:52:34.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fcade455-f3c2-4493-a4d8-5693e61c5075 2024-09-16T07:52:34.437 INFO:tasks.workunit.client.0.smithi162.stdout:add osd5 fcade455-f3c2-4493-a4d8-5693e61c5075 2024-09-16T07:52:34.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 fcade455-f3c2-4493-a4d8-5693e61c5075' 2024-09-16T07:52:34.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:52:34.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDC4+dmfF6rGxAA/nohKNcZDx92UK4dSWmMFQ== 2024-09-16T07:52:34.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDC4+dmfF6rGxAA/nohKNcZDx92UK4dSWmMFQ=="}' 2024-09-16T07:52:34.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fcade455-f3c2-4493-a4d8-5693e61c5075 -i td/test-erasure-eio/5/new.json 2024-09-16T07:52:35.318 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:52:35.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-16T07:52:35.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDC4+dmfF6rGxAA/nohKNcZDx92UK4dSWmMFQ== --osd-uuid fcade455-f3c2-4493-a4d8-5693e61c5075 2024-09-16T07:52:35.379 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:52:35.375+0000 7f22528b1880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:52:35.384 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:52:35.379+0000 7f22528b1880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:52:35.386 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:52:35.379+0000 7f22528b1880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:52:35.387 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:52:35.383+0000 7f22528b1880 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-16T07:52:35.387 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:52:35.383+0000 7f22528b1880 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-16T07:52:35.387 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:52:35.383+0000 7f22528b1880 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-16T07:52:35.564 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:52:35.559+0000 7f22528b1880 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-16T07:52:43.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-16T07:52:43.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:52:43.497 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd5 key to auth repository 2024-09-16T07:52:43.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-16T07:52:43.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:52:44.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-16T07:52:44.057 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.5 2024-09-16T07:52:44.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:52:44.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:52:44.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:52:44.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:52:44.088 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:52:44.083+0000 7fd4d0839880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:52:44.130 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:52:44.123+0000 7fd4d0839880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:52:44.188 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:52:44.127+0000 7fd4d0839880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:52:44.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-16T07:52:44.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:52:44.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-16T07:52:44.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:52:44.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:52:44.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:44.479 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:52:44.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:52:44.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:44.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:52:44.669 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:52:44.663+0000 7fd4d0839880 -1 Falling back to public interface 2024-09-16T07:52:44.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:45.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:45.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:45.951 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:52:45.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:52:45.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:45.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:52:46.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:47.296 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:52:47.291+0000 7fd4d0839880 -1 osd.5 0 log_to_monitors true 2024-09-16T07:52:47.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:47.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:47.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:52:47.366 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:52:47.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:47.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:52:48.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:49.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:49.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:49.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:52:49.145 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:52:49.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:49.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:52:49.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:50.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:50.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:50.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:52:50.759 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:52:50.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:50.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:52:51.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:52.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:52.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:52.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:52:52.313 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:52:52.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:52.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:52:52.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:53.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:53.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:53.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:52:53.750 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:52:53.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:53.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:52:54.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:55.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:55.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:55.293 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:52:55.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:52:55.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:55.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:52:55.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:56.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:56.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:56.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:52:56.786 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:52:56.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:56.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:52:57.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:58.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:58.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:58.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:52:58.292 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:52:58.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:58.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:52:58.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:52:59.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:52:59.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:52:59.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:52:59.817 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:52:59.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:52:59.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:53:00.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:53:01.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:53:01.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:01.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:53:01.347 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:53:01.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:01.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:53:02.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:53:03.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:53:03.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:03.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:53:03.211 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:53:03.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:03.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:53:04.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:53:05.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:53:05.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:05.393 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:53:05.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:53:05.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:05.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:53:06.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:53:07.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:53:07.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:07.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:53:07.357 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:53:07.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:07.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:53:07.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:53:08.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:53:08.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:08.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:53:08.881 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:53:08.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:08.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:53:09.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:53:10.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:53:10.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:10.859 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:53:10.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:53:10.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:10.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:53:11.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:53:12.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:53:12.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:12.895 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:53:12.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:53:12.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:12.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:53:13.366 INFO:tasks.workunit.client.0.smithi162.stdout:osd.5 up in weight 1 up_from 51 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2153992720,v1:127.0.0.1:6843/2153992720] [v2:127.0.0.1:6844/2153992720,v1:127.0.0.1:6845/2153992720] exists,up fcade455-f3c2-4493-a4d8-5693e61c5075 2024-09-16T07:53:13.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:53:13.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:53:13.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:53:13.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:53:13.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-16T07:53:13.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:53:13.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:53:13.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-09-16T07:53:13.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:53:13.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-16T07:53:13.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-16T07:53:13.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:53:13.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:53:13.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:53:13.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-16T07:53:13.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-16T07:53:13.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:53:13.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:53:13.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:53:13.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:53:13.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:53:13.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:53:13.371 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:53:13.371 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:53:13.371 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:53:13.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:53:13.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:53:13.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:53:13.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:53:13.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:53:13.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:53:13.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:53:13.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:53:13.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:53:13.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:53:13.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:53:13.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:53:13.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:53:13.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-16T07:53:13.374 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:53:13.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=88a78e7d-f13e-410d-a04e-2a1718954731 2024-09-16T07:53:13.374 INFO:tasks.workunit.client.0.smithi162.stdout:add osd6 88a78e7d-f13e-410d-a04e-2a1718954731 2024-09-16T07:53:13.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 88a78e7d-f13e-410d-a04e-2a1718954731' 2024-09-16T07:53:13.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:53:13.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDp4+dmzq29FxAAZuRD7ufWOzKlh4I/ipGMqQ== 2024-09-16T07:53:13.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDp4+dmzq29FxAAZuRD7ufWOzKlh4I/ipGMqQ=="}' 2024-09-16T07:53:13.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 88a78e7d-f13e-410d-a04e-2a1718954731 -i td/test-erasure-eio/6/new.json 2024-09-16T07:53:14.106 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:53:14.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-16T07:53:14.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDp4+dmzq29FxAAZuRD7ufWOzKlh4I/ipGMqQ== --osd-uuid 88a78e7d-f13e-410d-a04e-2a1718954731 2024-09-16T07:53:14.160 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:53:14.158+0000 7f5f38bd7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:53:14.223 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:53:14.222+0000 7f5f38bd7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:53:14.225 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:53:14.222+0000 7f5f38bd7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:53:14.226 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:53:14.226+0000 7f5f38bd7880 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-16T07:53:14.226 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:53:14.226+0000 7f5f38bd7880 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-16T07:53:14.226 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:53:14.226+0000 7f5f38bd7880 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-16T07:53:14.348 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:53:14.346+0000 7f5f38bd7880 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-16T07:53:21.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-16T07:53:21.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:53:21.621 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd6 key to auth repository 2024-09-16T07:53:21.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-09-16T07:53:21.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:53:22.198 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.6 2024-09-16T07:53:22.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-09-16T07:53:22.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:53:22.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:53:22.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:53:22.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:53:22.238 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:53:22.238+0000 7f6b18453880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:53:22.270 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:53:22.266+0000 7f6b18453880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:53:22.271 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:53:22.270+0000 7f6b18453880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:53:22.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-09-16T07:53:22.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:53:22.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-09-16T07:53:22.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:53:22.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:53:22.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:22.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:53:22.679 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:53:22.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:22.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:53:22.853 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:53:22.850+0000 7f6b18453880 -1 Falling back to public interface 2024-09-16T07:53:23.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:53:24.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:53:24.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:24.126 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:53:24.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:53:24.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:24.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:53:24.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:53:25.039 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:53:25.038+0000 7f6b18453880 -1 osd.6 0 log_to_monitors true 2024-09-16T07:53:25.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:53:25.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:25.703 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:53:25.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:53:25.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:25.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:53:26.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:53:27.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:53:27.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:27.360 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:53:27.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:53:27.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:27.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:53:28.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:53:29.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:53:29.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:29.013 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:53:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:53:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:53:29.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:53:30.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:53:30.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:30.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:53:30.513 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:53:30.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:30.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:53:31.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:53:32.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:53:32.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:32.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:53:32.140 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:53:32.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:32.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:53:32.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:53:33.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:53:33.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:33.651 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:53:33.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:53:33.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:33.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:53:34.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:53:35.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:53:35.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:35.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:53:35.197 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:53:35.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:35.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:53:35.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:53:36.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:53:36.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:36.643 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:53:36.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:53:36.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:36.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:53:37.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:53:38.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:53:38.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:38.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:53:38.149 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:53:38.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:38.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:53:38.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:53:39.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:53:39.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:39.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:53:39.765 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:53:39.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:39.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:53:40.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:53:41.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:53:41.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:41.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:53:41.633 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:53:41.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:41.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:53:42.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:53:43.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:53:43.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:43.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:53:43.428 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:53:43.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:43.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:53:44.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:53:45.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:53:45.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:45.156 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:53:45.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:53:45.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:45.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:53:46.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:53:47.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:53:47.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:47.358 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:53:47.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:53:47.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:47.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:53:48.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:53:49.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:53:49.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:49.405 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:53:49.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:53:49.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:49.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:53:50.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:53:51.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:53:51.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:53:51.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:53:51.236 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:53:51.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:53:51.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:53:51.671 INFO:tasks.workunit.client.0.smithi162.stdout:osd.6 up in weight 1 up_from 61 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1041054711,v1:127.0.0.1:6851/1041054711] [v2:127.0.0.1:6852/1041054711,v1:127.0.0.1:6853/1041054711] exists,up 88a78e7d-f13e-410d-a04e-2a1718954731 2024-09-16T07:53:51.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:53:51.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:53:51.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:53:51.672 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-16T07:53:51.672 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T07:53:51.672 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T07:53:51.673 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:53:51.673 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:53:51.673 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:53:51.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.0.asok 2024-09-16T07:53:51.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-16T07:53:51.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.0.asok log flush 2024-09-16T07:53:51.761 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:53:51.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-16T07:53:51.780 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:49:39.046+0000 7f021ec89880 0 load: jerasure load: lrc 2024-09-16T07:53:51.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:457: TEST_ec_recovery_multiple_objects: CEPH_ARGS='--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:53:51.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:459: TEST_ec_recovery_multiple_objects: local poolname=pool-jerasure 2024-09-16T07:53:51.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:460: TEST_ec_recovery_multiple_objects: create_erasure_coded_pool pool-jerasure 3 2 2024-09-16T07:53:51.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T07:53:51.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-16T07:53:51.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-16T07:53:51.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-16T07:53:51.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-16T07:53:51.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-16T07:53:51.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-16T07:53:52.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-16T07:53:52.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-16T07:53:53.984 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-16T07:53:53.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T07:53:55.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-16T07:53:55.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T07:53:55.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T07:53:55.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T07:53:55.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T07:53:55.001 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T07:53:55.002 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T07:53:55.002 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T07:53:55.003 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T07:53:55.003 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T07:53:55.003 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T07:53:55.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T07:53:55.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T07:53:55.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T07:53:55.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T07:53:55.218 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T07:53:55.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T07:53:55.691 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:53:55.692 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:53:55.692 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:53:55.692 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:53:55.692 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:53:55.692 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-16T07:53:55.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T07:53:55.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:53:55.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T07:53:55.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803836 2024-09-16T07:53:55.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803836 2024-09-16T07:53:55.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803836' 2024-09-16T07:53:55.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:53:55.856 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T07:53:56.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=55834574900 2024-09-16T07:53:56.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 55834574900 2024-09-16T07:53:56.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803836 1-55834574900' 2024-09-16T07:53:56.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:53:56.043 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T07:53:56.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378669 2024-09-16T07:53:56.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378669 2024-09-16T07:53:56.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803836 1-55834574900 2-81604378669' 2024-09-16T07:53:56.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:53:56.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T07:53:56.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=124554051627 2024-09-16T07:53:56.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 124554051627 2024-09-16T07:53:56.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803836 1-55834574900 2-81604378669 3-124554051627' 2024-09-16T07:53:56.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:53:56.359 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T07:53:56.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=171798691870 2024-09-16T07:53:56.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 171798691870 2024-09-16T07:53:56.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803836 1-55834574900 2-81604378669 3-124554051627 4-171798691870' 2024-09-16T07:53:56.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:53:56.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T07:53:56.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=219043332121 2024-09-16T07:53:56.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 219043332121 2024-09-16T07:53:56.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803836 1-55834574900 2-81604378669 3-124554051627 4-171798691870 5-219043332121' 2024-09-16T07:53:56.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:53:56.666 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-16T07:53:56.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=261993005073 2024-09-16T07:53:56.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 261993005073 2024-09-16T07:53:56.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803836 1-55834574900 2-81604378669 3-124554051627 4-171798691870 5-219043332121 6-261993005073' 2024-09-16T07:53:56.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:53:56.815 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803836 2024-09-16T07:53:56.815 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:53:56.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T07:53:56.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803836 2024-09-16T07:53:56.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:53:56.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803836 2024-09-16T07:53:56.820 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803836 2024-09-16T07:53:56.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803836' 2024-09-16T07:53:56.821 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:53:57.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803834 -lt 25769803836 2024-09-16T07:53:57.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T07:53:58.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T07:53:58.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:53:58.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803837 -lt 25769803836 2024-09-16T07:53:58.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:53:58.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-55834574900 2024-09-16T07:53:58.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:53:58.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T07:53:58.797 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-55834574900 2024-09-16T07:53:58.797 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:53:58.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=55834574900 2024-09-16T07:53:58.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 55834574900' 2024-09-16T07:53:58.800 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574900 2024-09-16T07:53:58.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T07:53:59.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 55834574900 -lt 55834574900 2024-09-16T07:53:59.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:53:59.218 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378669 2024-09-16T07:53:59.218 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:53:59.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T07:53:59.220 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:53:59.220 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378669 2024-09-16T07:53:59.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378669 2024-09-16T07:53:59.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378669' 2024-09-16T07:53:59.222 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 81604378669 2024-09-16T07:53:59.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T07:53:59.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378669 -lt 81604378669 2024-09-16T07:53:59.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:53:59.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-124554051627 2024-09-16T07:53:59.674 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:53:59.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T07:53:59.676 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-124554051627 2024-09-16T07:53:59.677 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:53:59.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=124554051627 2024-09-16T07:53:59.679 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 124554051627 2024-09-16T07:53:59.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 124554051627' 2024-09-16T07:53:59.679 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T07:54:00.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 124554051628 -lt 124554051627 2024-09-16T07:54:00.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:54:00.108 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-171798691870 2024-09-16T07:54:00.108 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:54:00.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T07:54:00.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-171798691870 2024-09-16T07:54:00.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:54:00.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=171798691870 2024-09-16T07:54:00.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 171798691870' 2024-09-16T07:54:00.113 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 171798691870 2024-09-16T07:54:00.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T07:54:00.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 171798691870 -lt 171798691870 2024-09-16T07:54:00.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:54:00.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-219043332121 2024-09-16T07:54:00.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:54:00.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T07:54:00.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:54:00.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-219043332121 2024-09-16T07:54:00.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=219043332121 2024-09-16T07:54:00.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 219043332121' 2024-09-16T07:54:00.609 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 219043332121 2024-09-16T07:54:00.609 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T07:54:01.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 219043332122 -lt 219043332121 2024-09-16T07:54:01.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:54:01.020 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 6-261993005073 2024-09-16T07:54:01.021 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:54:01.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=6 2024-09-16T07:54:01.023 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 6-261993005073 2024-09-16T07:54:01.023 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:54:01.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=261993005073 2024-09-16T07:54:01.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.6 seq 261993005073' 2024-09-16T07:54:01.025 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 261993005073 2024-09-16T07:54:01.026 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-16T07:54:01.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 261993005073 -lt 261993005073 2024-09-16T07:54:01.462 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T07:54:01.462 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:54:01.462 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:54:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T07:54:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T07:54:01.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T07:54:01.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T07:54:01.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T07:54:01.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T07:54:01.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T07:54:01.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T07:54:02.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T07:54:02.422 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T07:54:02.423 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:54:02.423 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:54:02.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T07:54:02.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T07:54:02.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T07:54:02.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:462: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test1 2024-09-16T07:54:02.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-16T07:54:02.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-16T07:54:02.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2024-09-16T07:54:02.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:54:02.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-16T07:54:02.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:54:02.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-16T07:54:02.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:54:02.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-16T07:54:02.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:54:02.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-16T07:54:02.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2024-09-16T07:54:03.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:463: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test2 2024-09-16T07:54:03.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-16T07:54:03.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-16T07:54:03.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2024-09-16T07:54:03.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:54:03.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-16T07:54:03.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:54:03.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-16T07:54:03.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:54:03.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-16T07:54:03.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:54:03.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-16T07:54:03.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2024-09-16T07:54:03.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:464: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test3 2024-09-16T07:54:03.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-16T07:54:03.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-16T07:54:03.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2024-09-16T07:54:03.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:54:03.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-16T07:54:03.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:54:03.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-16T07:54:03.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:54:03.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-16T07:54:03.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:54:03.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-16T07:54:03.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2024-09-16T07:54:03.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:466: TEST_ec_recovery_multiple_objects: ceph osd out 0 2024-09-16T07:54:04.635 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 is already out. 2024-09-16T07:54:04.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:469: TEST_ec_recovery_multiple_objects: wait_for_clean 2024-09-16T07:54:04.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T07:54:04.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T07:54:04.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T07:54:04.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T07:54:04.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T07:54:04.652 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T07:54:04.652 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T07:54:04.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T07:54:04.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T07:54:04.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T07:54:04.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T07:54:04.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T07:54:04.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T07:54:04.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T07:54:04.851 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T07:54:05.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T07:54:05.546 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:54:05.546 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:54:05.546 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:54:05.546 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:54:05.546 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:54:05.546 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-16T07:54:05.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T07:54:05.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:54:05.547 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T07:54:05.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803840 2024-09-16T07:54:05.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803840 2024-09-16T07:54:05.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803840' 2024-09-16T07:54:05.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:54:05.703 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T07:54:05.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=55834574904 2024-09-16T07:54:05.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 55834574904 2024-09-16T07:54:05.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803840 1-55834574904' 2024-09-16T07:54:05.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:54:05.988 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T07:54:06.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378673 2024-09-16T07:54:06.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378673 2024-09-16T07:54:06.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803840 1-55834574904 2-81604378673' 2024-09-16T07:54:06.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:54:06.145 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T07:54:06.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=124554051631 2024-09-16T07:54:06.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 124554051631 2024-09-16T07:54:06.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803840 1-55834574904 2-81604378673 3-124554051631' 2024-09-16T07:54:06.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:54:06.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T07:54:06.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=171798691874 2024-09-16T07:54:06.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 171798691874 2024-09-16T07:54:06.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803840 1-55834574904 2-81604378673 3-124554051631 4-171798691874' 2024-09-16T07:54:06.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:54:06.492 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T07:54:06.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=219043332125 2024-09-16T07:54:06.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 219043332125 2024-09-16T07:54:06.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803840 1-55834574904 2-81604378673 3-124554051631 4-171798691874 5-219043332125' 2024-09-16T07:54:06.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:54:06.729 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-16T07:54:06.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=261993005077 2024-09-16T07:54:06.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 261993005077 2024-09-16T07:54:06.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803840 1-55834574904 2-81604378673 3-124554051631 4-171798691874 5-219043332125 6-261993005077' 2024-09-16T07:54:06.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:54:06.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803840 2024-09-16T07:54:06.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:54:06.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T07:54:06.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803840 2024-09-16T07:54:06.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:54:06.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803840 2024-09-16T07:54:06.890 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803840 2024-09-16T07:54:06.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803840' 2024-09-16T07:54:06.890 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:54:07.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803840 -lt 25769803840 2024-09-16T07:54:07.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:54:07.322 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-55834574904 2024-09-16T07:54:07.322 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:54:07.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T07:54:07.324 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-55834574904 2024-09-16T07:54:07.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:54:07.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=55834574904 2024-09-16T07:54:07.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 55834574904' 2024-09-16T07:54:07.327 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574904 2024-09-16T07:54:07.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T07:54:07.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 55834574902 -lt 55834574904 2024-09-16T07:54:07.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T07:54:08.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T07:54:08.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T07:54:09.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 55834574904 -lt 55834574904 2024-09-16T07:54:09.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:54:09.183 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378673 2024-09-16T07:54:09.183 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:54:09.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T07:54:09.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378673 2024-09-16T07:54:09.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:54:09.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378673 2024-09-16T07:54:09.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378673' 2024-09-16T07:54:09.188 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 81604378673 2024-09-16T07:54:09.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T07:54:09.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378673 -lt 81604378673 2024-09-16T07:54:09.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:54:09.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-124554051631 2024-09-16T07:54:09.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:54:09.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T07:54:09.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-124554051631 2024-09-16T07:54:09.715 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:54:09.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=124554051631 2024-09-16T07:54:09.717 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 124554051631 2024-09-16T07:54:09.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 124554051631' 2024-09-16T07:54:09.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T07:54:10.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 124554051632 -lt 124554051631 2024-09-16T07:54:10.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:54:10.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-171798691874 2024-09-16T07:54:10.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:54:10.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T07:54:10.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-171798691874 2024-09-16T07:54:10.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:54:10.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=171798691874 2024-09-16T07:54:10.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 171798691874' 2024-09-16T07:54:10.152 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 171798691874 2024-09-16T07:54:10.153 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T07:54:10.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 171798691874 -lt 171798691874 2024-09-16T07:54:10.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:54:10.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-219043332125 2024-09-16T07:54:10.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:54:10.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T07:54:10.624 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-219043332125 2024-09-16T07:54:10.624 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:54:10.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=219043332125 2024-09-16T07:54:10.626 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 219043332125 2024-09-16T07:54:10.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 219043332125' 2024-09-16T07:54:10.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T07:54:11.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 219043332126 -lt 219043332125 2024-09-16T07:54:11.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:54:11.080 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 6-261993005077 2024-09-16T07:54:11.080 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:54:11.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=6 2024-09-16T07:54:11.083 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 6-261993005077 2024-09-16T07:54:11.083 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:54:11.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=261993005077 2024-09-16T07:54:11.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.6 seq 261993005077' 2024-09-16T07:54:11.086 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 261993005077 2024-09-16T07:54:11.086 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-16T07:54:11.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 261993005077 -lt 261993005077 2024-09-16T07:54:11.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T07:54:11.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:54:11.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:54:12.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T07:54:12.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T07:54:12.151 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T07:54:12.151 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T07:54:12.151 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T07:54:12.151 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T07:54:12.151 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T07:54:12.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T07:54:12.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T07:54:12.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T07:54:12.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:54:12.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:54:13.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T07:54:13.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T07:54:13.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T07:54:13.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:471: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test1 2024-09-16T07:54:13.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T07:54:13.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T07:54:13.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2024-09-16T07:54:13.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-16T07:54:13.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-16T07:54:13.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2024-09-16T07:54:13.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-16T07:54:13.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-16T07:54:13.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:472: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test2 2024-09-16T07:54:13.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T07:54:13.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T07:54:13.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2024-09-16T07:54:13.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-16T07:54:13.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-16T07:54:13.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2024-09-16T07:54:13.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-16T07:54:13.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-16T07:54:13.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:473: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test3 2024-09-16T07:54:13.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T07:54:13.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T07:54:13.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2024-09-16T07:54:13.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-16T07:54:13.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-16T07:54:13.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2024-09-16T07:54:13.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-16T07:54:13.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-16T07:54:13.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:475: TEST_ec_recovery_multiple_objects: delete_erasure_coded_pool pool-jerasure 2024-09-16T07:54:13.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T07:54:13.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-16T07:54:14.091 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-16T07:54:14.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-16T07:54:14.831 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-16T07:54:14.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-16T07:54:14.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T07:54:14.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:54:14.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T07:54:14.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:54:14.842 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:54:14.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:54:14.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:54:14.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:54:15.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:54:15.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:54:15.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:54:15.167 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:54:15.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:54:15.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:54:15.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:54:15.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:54:15.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:54:15.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:54:15.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:54:15.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:54:15.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:54:15.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T07:54:15.216 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:54:15.216 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:54:15.216 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:54:15.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T07:54:15.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:54:15.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:54:15.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-16T07:54:15.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-16T07:54:15.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-16T07:54:15.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-16T07:54:15.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T07:54:15.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:54:15.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T07:54:15.220 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:54:15.220 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:54:15.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:54:15.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:54:15.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:54:15.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:54:15.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:54:15.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:54:15.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:54:15.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:54:15.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:54:15.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:54:15.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:54:15.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:54:15.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:54:15.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:54:15.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:54:15.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:54:15.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T07:54:15.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:54:15.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:54:15.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:54:15.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T07:54:15.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:54:15.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:54:15.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-16T07:54:15.243 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T07:54:15.243 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:54:15.243 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:54:15.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79228 2024-09-16T07:54:15.246 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T07:54:15.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T07:54:15.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T07:54:15.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-16T07:54:15.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-16T07:54:15.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-16T07:54:15.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T07:54:15.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T07:54:15.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T07:54:15.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-16T07:54:15.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-16T07:54:15.703 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T07:54:15.704 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:54:15.704 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:54:15.704 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:54:15.704 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:54:15.704 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:54:15.705 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:54:15.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T07:54:15.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T07:54:15.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T07:54:15.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:54:15.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:54:15.915 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T07:54:15.915 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T07:54:15.916 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:54:15.916 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:54:15.916 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:54:15.916 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:54:15.917 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:54:15.917 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:54:15.917 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T07:54:15.918 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:54:15.918 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get fsid 2024-09-16T07:54:16.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T07:54:16.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:54:16.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:54:16.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T07:54:16.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T07:54:16.026 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:54:16.026 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:54:16.026 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:54:16.026 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:54:16.026 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:54:16.027 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:54:16.027 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T07:54:16.027 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:54:16.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get mon_host 2024-09-16T07:54:16.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-16T07:54:16.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-16T07:54:16.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T07:54:16.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T07:54:16.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T07:54:16.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-16T07:54:16.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T07:54:17.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T07:54:17.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:54:17.029 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:54:17.029 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:54:17.029 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:54:17.029 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:54:17.029 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:54:17.030 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:54:17.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:54:17.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-16T07:54:17.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T07:54:17.433 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-16T07:54:17.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T07:54:18.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-16T07:54:18.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:54:18.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:54:18.446 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:54:18.446 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:54:18.446 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:54:18.446 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T07:54:18.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-16T07:54:18.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-mon.a.asok log flush 2024-09-16T07:54:18.530 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:54:18.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-16T07:54:18.542 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:54:15.757+0000 7f8fe11afd40 0 load: jerasure load: lrc 2024-09-16T07:54:18.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects_eio td/test-erasure-eio 2024-09-16T07:54:18.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:480: TEST_ec_recovery_multiple_objects_eio: local dir=td/test-erasure-eio 2024-09-16T07:54:18.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:481: TEST_ec_recovery_multiple_objects_eio: local objname=myobject 2024-09-16T07:54:18.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:483: TEST_ec_recovery_multiple_objects_eio: ORIG_ARGS='--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:54:18.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:484: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-16T07:54:18.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:485: TEST_ec_recovery_multiple_objects_eio: setup_osds 7 2024-09-16T07:54:18.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-09-16T07:54:18.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-16T07:54:18.544 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-09-16T07:54:18.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-09-16T07:54:18.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:54:18.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-16T07:54:18.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:54:18.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:54:18.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T07:54:18.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:54:18.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-16T07:54:18.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-16T07:54:18.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:54:18.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:54:18.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:54:18.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-16T07:54:18.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-16T07:54:18.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:54:18.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:54:18.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:54:18.550 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:54:18.550 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:54:18.550 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:54:18.550 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:54:18.550 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:54:18.550 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:54:18.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:54:18.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:54:18.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:54:18.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:54:18.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:54:18.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:54:18.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:54:18.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:54:18.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:54:18.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:54:18.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:54:18.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:54:18.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:54:18.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-16T07:54:18.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:54:18.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=edb12b99-88d6-4504-851d-81d3521afd4a 2024-09-16T07:54:18.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 edb12b99-88d6-4504-851d-81d3521afd4a' 2024-09-16T07:54:18.553 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 edb12b99-88d6-4504-851d-81d3521afd4a 2024-09-16T07:54:18.553 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:54:18.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAq5OdmE/OSIhAA+QBr9CbRB5QWiQmbrBGGTQ== 2024-09-16T07:54:18.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAq5OdmE/OSIhAA+QBr9CbRB5QWiQmbrBGGTQ=="}' 2024-09-16T07:54:18.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new edb12b99-88d6-4504-851d-81d3521afd4a -i td/test-erasure-eio/0/new.json 2024-09-16T07:54:18.958 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:54:18.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-16T07:54:18.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAq5OdmE/OSIhAA+QBr9CbRB5QWiQmbrBGGTQ== --osd-uuid edb12b99-88d6-4504-851d-81d3521afd4a 2024-09-16T07:54:19.011 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:54:19.009+0000 7fa5af44c880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:54:19.014 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:54:19.013+0000 7fa5af44c880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:54:19.016 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:54:19.013+0000 7fa5af44c880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:54:19.016 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:54:19.013+0000 7fa5af44c880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T07:54:19.017 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:54:19.013+0000 7fa5af44c880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T07:54:19.017 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:54:19.013+0000 7fa5af44c880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T07:54:19.102 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:54:19.097+0000 7fa5af44c880 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-16T07:54:24.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-16T07:54:24.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:54:24.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T07:54:24.874 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T07:54:24.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:54:25.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T07:54:25.442 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T07:54:25.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:54:25.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:54:25.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:54:25.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:54:25.481 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:54:25.477+0000 7ffb7e778880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:54:25.556 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:54:25.553+0000 7ffb7e778880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:54:25.557 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:54:25.553+0000 7ffb7e778880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:54:25.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T07:54:25.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:54:25.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T07:54:25.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:54:25.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:54:25.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:54:25.873 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:54:25.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:54:25.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:54:25.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:54:26.151 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:54:26.149+0000 7ffb7e778880 -1 Falling back to public interface 2024-09-16T07:54:26.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:54:27.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:54:27.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:54:27.299 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:54:27.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:54:27.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:54:27.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:54:27.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:54:28.725 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:54:28.721+0000 7ffb7e778880 -1 osd.0 0 log_to_monitors true 2024-09-16T07:54:28.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:54:28.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:54:28.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:54:28.728 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:54:28.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:54:28.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:54:29.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:54:30.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:54:30.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:54:30.315 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:54:30.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:54:30.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:54:30.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:54:30.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:54:31.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:54:31.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:54:31.785 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:54:31.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:54:31.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:54:31.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:54:32.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:54:33.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:54:33.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:54:33.225 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:54:33.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:54:33.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:54:33.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:54:33.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:54:34.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:54:34.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:54:34.794 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:54:34.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:54:34.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:54:34.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:54:35.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:54:36.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:54:36.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:54:36.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:54:36.239 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:54:36.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:54:36.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:54:36.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:54:37.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:54:37.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:54:37.735 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:54:37.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:54:37.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:54:37.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:54:38.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:54:39.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:54:39.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:54:39.185 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:54:39.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:54:39.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:54:39.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:54:39.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:54:40.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:54:40.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:54:40.758 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:54:40.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:54:40.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:54:40.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:54:41.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:54:42.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:54:42.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:54:42.201 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:54:42.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:54:42.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:54:42.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:54:42.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:54:43.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:54:43.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:54:43.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:54:43.946 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:54:43.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:54:43.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:54:45.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:54:46.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:54:46.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:54:46.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:54:46.103 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:54:46.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:54:46.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:54:46.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:54:47.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:54:47.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:54:47.825 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:54:47.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:54:47.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:54:47.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:54:48.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:54:49.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:54:49.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:54:49.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:54:49.782 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:54:49.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:54:49.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:54:50.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:54:51.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:54:51.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:54:51.298 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:54:51.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:54:51.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:54:51.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:54:51.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:54:52.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:54:52.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:54:52.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:54:52.906 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:54:52.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:54:52.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:54:53.455 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3350288965,v1:127.0.0.1:6803/3350288965] [v2:127.0.0.1:6804/3350288965,v1:127.0.0.1:6805/3350288965] exists,up edb12b99-88d6-4504-851d-81d3521afd4a 2024-09-16T07:54:53.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:54:53.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:54:53.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:54:53.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:54:53.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-16T07:54:53.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:54:53.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:54:53.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T07:54:53.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:54:53.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-16T07:54:53.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-16T07:54:53.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:54:53.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:54:53.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:54:53.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-16T07:54:53.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-16T07:54:53.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:54:53.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:54:53.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:54:53.459 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:54:53.459 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:54:53.459 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:54:53.461 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:54:53.461 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:54:53.461 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:54:53.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:54:53.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:54:53.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:54:53.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:54:53.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:54:53.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:54:53.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:54:53.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:54:53.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:54:53.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:54:53.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:54:53.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:54:53.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:54:53.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-16T07:54:53.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:54:53.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c644e66f-e9d0-41f0-b206-218c366bd997 2024-09-16T07:54:53.464 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 c644e66f-e9d0-41f0-b206-218c366bd997 2024-09-16T07:54:53.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c644e66f-e9d0-41f0-b206-218c366bd997' 2024-09-16T07:54:53.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:54:53.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBN5OdminwrHRAA2xPV7XvsXSvyyvL1DrcE6Q== 2024-09-16T07:54:53.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBN5OdminwrHRAA2xPV7XvsXSvyyvL1DrcE6Q=="}' 2024-09-16T07:54:53.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c644e66f-e9d0-41f0-b206-218c366bd997 -i td/test-erasure-eio/1/new.json 2024-09-16T07:54:54.302 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:54:54.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-16T07:54:54.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBN5OdminwrHRAA2xPV7XvsXSvyyvL1DrcE6Q== --osd-uuid c644e66f-e9d0-41f0-b206-218c366bd997 2024-09-16T07:54:54.355 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:54:54.352+0000 7f3a5311f880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:54:54.399 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:54:54.396+0000 7f3a5311f880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:54:54.401 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:54:54.396+0000 7f3a5311f880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:54:54.402 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:54:54.400+0000 7f3a5311f880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T07:54:54.403 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:54:54.400+0000 7f3a5311f880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T07:54:54.403 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:54:54.400+0000 7f3a5311f880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T07:54:54.491 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:54:54.488+0000 7f3a5311f880 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-16T07:55:00.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-16T07:55:00.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:55:00.761 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-16T07:55:00.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T07:55:00.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:55:01.385 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-16T07:55:01.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T07:55:01.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:55:01.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:55:01.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:55:01.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:55:01.422 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:55:01.420+0000 7f414ce07880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:55:01.450 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:55:01.448+0000 7f414ce07880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:55:01.452 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:55:01.448+0000 7f414ce07880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:55:01.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T07:55:01.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:55:01.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T07:55:01.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:55:01.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:55:01.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:01.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:55:01.819 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:55:01.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:01.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:55:01.988 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:55:01.984+0000 7f414ce07880 -1 Falling back to public interface 2024-09-16T07:55:02.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:03.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:03.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:03.270 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:55:03.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:55:03.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:03.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:55:03.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:04.650 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:55:04.648+0000 7f414ce07880 -1 osd.1 0 log_to_monitors true 2024-09-16T07:55:04.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:04.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:04.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:55:04.686 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:55:04.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:04.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:55:05.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:06.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:06.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:06.152 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:55:06.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:55:06.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:06.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:55:06.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:07.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:07.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:07.723 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:55:07.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:55:07.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:07.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:55:08.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:09.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:09.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:09.212 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:55:09.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:55:09.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:09.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:55:09.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:10.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:10.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:10.817 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:55:10.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:55:10.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:10.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:55:11.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:12.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:12.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:12.252 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:55:12.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:55:12.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:12.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:55:12.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:13.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:13.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:13.800 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:55:13.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:55:13.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:13.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:55:14.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:15.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:15.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:15.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:55:15.268 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:55:15.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:15.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:55:15.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:16.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:16.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:16.904 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:55:16.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:55:16.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:16.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:55:17.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:18.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:18.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:18.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:55:18.453 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:55:18.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:18.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:55:19.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:20.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:20.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:20.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:55:20.222 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:55:20.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:20.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:55:21.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:22.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:22.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:22.049 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:55:22.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:55:22.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:22.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:55:22.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:23.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:23.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:23.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:55:23.972 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:55:23.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:23.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:55:25.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:26.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:26.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:26.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:55:26.225 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:55:26.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:26.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:55:27.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:28.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:28.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:28.041 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:55:28.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:55:28.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:28.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:55:28.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:29.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:29.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:29.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:55:29.589 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:55:29.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:29.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:55:30.189 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/522964285,v1:127.0.0.1:6811/522964285] [v2:127.0.0.1:6812/522964285,v1:127.0.0.1:6813/522964285] exists,up c644e66f-e9d0-41f0-b206-218c366bd997 2024-09-16T07:55:30.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:55:30.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:55:30.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:55:30.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:55:30.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-16T07:55:30.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:55:30.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:55:30.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T07:55:30.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:55:30.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-16T07:55:30.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-16T07:55:30.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:55:30.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:55:30.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:55:30.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-16T07:55:30.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-16T07:55:30.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:55:30.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:55:30.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:55:30.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:55:30.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:55:30.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:55:30.195 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:55:30.195 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:55:30.196 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:55:30.196 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:55:30.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:55:30.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:55:30.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:55:30.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:55:30.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:55:30.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:55:30.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:55:30.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:55:30.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:55:30.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:55:30.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:55:30.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:55:30.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-16T07:55:30.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:55:30.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4800dea9-0174-45e5-9678-5d79ebbb3430 2024-09-16T07:55:30.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4800dea9-0174-45e5-9678-5d79ebbb3430' 2024-09-16T07:55:30.198 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 4800dea9-0174-45e5-9678-5d79ebbb3430 2024-09-16T07:55:30.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:55:30.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBy5OdmyKOWDRAADsnFbFU943obRLc+O99wXQ== 2024-09-16T07:55:30.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBy5OdmyKOWDRAADsnFbFU943obRLc+O99wXQ=="}' 2024-09-16T07:55:30.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4800dea9-0174-45e5-9678-5d79ebbb3430 -i td/test-erasure-eio/2/new.json 2024-09-16T07:55:30.909 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:55:30.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-16T07:55:30.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBy5OdmyKOWDRAADsnFbFU943obRLc+O99wXQ== --osd-uuid 4800dea9-0174-45e5-9678-5d79ebbb3430 2024-09-16T07:55:30.973 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:55:30.967+0000 7f2e09821880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:55:30.975 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:55:30.971+0000 7f2e09821880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:55:30.977 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:55:30.971+0000 7f2e09821880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:55:30.978 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:55:30.975+0000 7f2e09821880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T07:55:30.978 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:55:30.975+0000 7f2e09821880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T07:55:30.979 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:55:30.975+0000 7f2e09821880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T07:55:31.045 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:55:31.039+0000 7f2e09821880 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-16T07:55:37.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-16T07:55:37.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:55:37.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T07:55:37.641 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-16T07:55:37.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:55:38.208 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-16T07:55:38.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T07:55:38.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:55:38.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:55:38.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:55:38.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:55:38.248 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:55:38.243+0000 7fea130d1880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:55:38.271 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:55:38.267+0000 7fea130d1880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:55:38.273 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:55:38.267+0000 7fea130d1880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:55:38.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T07:55:38.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:55:38.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T07:55:38.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:55:38.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:55:38.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:38.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:55:38.709 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:55:38.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:38.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:55:38.828 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:55:38.823+0000 7fea130d1880 -1 Falling back to public interface 2024-09-16T07:55:39.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:40.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:40.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:40.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:55:40.164 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:55:40.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:40.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:55:40.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:41.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:41.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:41.585 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:55:41.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:55:41.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:55:41.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:41.606 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:55:41.603+0000 7fea130d1880 -1 osd.2 0 log_to_monitors true 2024-09-16T07:55:42.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:43.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:43.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:43.078 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:55:43.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:55:43.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:43.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:55:43.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:44.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:44.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:44.554 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:55:44.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:55:44.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:44.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:55:45.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:46.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:46.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:46.073 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:55:46.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:55:46.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:46.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:55:46.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:47.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:47.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:47.518 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:55:47.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:55:47.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:47.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:55:47.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:48.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:48.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:48.975 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:55:48.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:55:48.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:48.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:55:49.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:50.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:50.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:50.577 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:55:50.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:55:50.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:50.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:55:51.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:52.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:52.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:52.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:55:52.121 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:55:52.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:52.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:55:52.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:53.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:53.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:53.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:55:53.606 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:55:53.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:53.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:55:54.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:55.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:55.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:55.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:55:55.033 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:55:55.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:55.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:55:55.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:56.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:56.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:56.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:55:56.977 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:55:56.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:56.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:55:57.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:55:58.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:55:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:55:58.627 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:55:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:55:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:55:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:55:59.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:00.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:00.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:00.628 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:56:00.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:56:00.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:00.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:56:01.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:02.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:02.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:02.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:56:02.197 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:56:02.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:02.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:56:02.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:03.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:03.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:03.858 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:56:03.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:56:03.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:03.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:56:04.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:06.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:06.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:06.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:56:06.001 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:56:06.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:06.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T07:56:06.497 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4258261397,v1:127.0.0.1:6819/4258261397] [v2:127.0.0.1:6820/4258261397,v1:127.0.0.1:6821/4258261397] exists,up 4800dea9-0174-45e5-9678-5d79ebbb3430 2024-09-16T07:56:06.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:56:06.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:56:06.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:56:06.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:56:06.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-16T07:56:06.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:56:06.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:56:06.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T07:56:06.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:56:06.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-16T07:56:06.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-16T07:56:06.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:56:06.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:56:06.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:56:06.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-16T07:56:06.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-16T07:56:06.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:56:06.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:56:06.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:56:06.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:56:06.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:56:06.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:56:06.502 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:56:06.502 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:56:06.502 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:56:06.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:56:06.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:56:06.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:56:06.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:56:06.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:56:06.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:56:06.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:56:06.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:56:06.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:56:06.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:56:06.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:56:06.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:56:06.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:56:06.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-16T07:56:06.504 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:56:06.505 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 0b30b8ef-776f-4ca2-9e01-c3fd1a3ff04c 2024-09-16T07:56:06.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0b30b8ef-776f-4ca2-9e01-c3fd1a3ff04c 2024-09-16T07:56:06.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 0b30b8ef-776f-4ca2-9e01-c3fd1a3ff04c' 2024-09-16T07:56:06.505 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:56:06.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCW5OdmNZnCHxAAc8v0V+82w+1tN5Vl0BJ7EQ== 2024-09-16T07:56:06.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCW5OdmNZnCHxAAc8v0V+82w+1tN5Vl0BJ7EQ=="}' 2024-09-16T07:56:06.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0b30b8ef-776f-4ca2-9e01-c3fd1a3ff04c -i td/test-erasure-eio/3/new.json 2024-09-16T07:56:08.081 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:56:08.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-16T07:56:08.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCW5OdmNZnCHxAAc8v0V+82w+1tN5Vl0BJ7EQ== --osd-uuid 0b30b8ef-776f-4ca2-9e01-c3fd1a3ff04c 2024-09-16T07:56:08.140 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:56:08.135+0000 7f2a5b5c2880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:56:08.189 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:56:08.183+0000 7f2a5b5c2880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:56:08.191 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:56:08.187+0000 7f2a5b5c2880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:56:08.192 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:56:08.187+0000 7f2a5b5c2880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T07:56:08.192 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:56:08.187+0000 7f2a5b5c2880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T07:56:08.192 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:56:08.187+0000 7f2a5b5c2880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T07:56:08.296 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:56:08.295+0000 7f2a5b5c2880 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-16T07:56:14.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-16T07:56:14.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:56:14.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T07:56:14.761 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-16T07:56:14.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:56:15.335 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-16T07:56:15.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T07:56:15.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:56:15.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:56:15.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:56:15.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:56:15.376 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:56:15.371+0000 7f902ec51880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:56:15.378 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:56:15.375+0000 7f902ec51880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:56:15.380 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:56:15.375+0000 7f902ec51880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:56:15.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T07:56:15.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:56:15.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T07:56:15.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:56:15.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:56:15.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:15.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:56:15.783 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:56:15.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:15.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:56:15.924 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:56:15.919+0000 7f902ec51880 -1 Falling back to public interface 2024-09-16T07:56:16.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:17.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:17.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:17.229 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:56:17.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:56:17.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:17.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:56:17.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:18.542 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:56:18.539+0000 7f902ec51880 -1 osd.3 0 log_to_monitors true 2024-09-16T07:56:18.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:18.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:18.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:56:18.665 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:56:18.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:18.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:56:19.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:20.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:20.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:20.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:56:20.100 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:56:20.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:20.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:56:21.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:22.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:22.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:22.072 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:56:22.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:56:22.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:22.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:56:22.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:23.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:23.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:23.585 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:56:23.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:56:23.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:23.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:56:24.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:25.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:25.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:25.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:56:25.121 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:56:25.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:25.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:56:25.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:26.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:26.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:26.607 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:56:26.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:56:26.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:26.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:56:27.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:28.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:28.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:28.164 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:56:28.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:56:28.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:28.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:56:28.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:29.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:29.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:29.770 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:56:29.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:56:29.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:29.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:56:30.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:31.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:31.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:31.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:56:31.260 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:56:31.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:31.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:56:31.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:32.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:32.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:32.744 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:56:32.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:56:32.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:32.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:56:33.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:34.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:34.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:34.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:56:34.295 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:56:34.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:34.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:56:34.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:35.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:35.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:35.915 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:56:35.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:56:35.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:35.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:56:36.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:37.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:37.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:37.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:56:37.977 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:56:37.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:37.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:56:39.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:40.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:40.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:40.099 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:56:40.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:56:40.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:40.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:56:40.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:41.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:41.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:41.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:56:41.616 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:56:41.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:41.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:56:42.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:43.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:43.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:43.336 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:56:43.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:56:43.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:43.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:56:43.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:44.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:44.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:44.783 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-16T07:56:44.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T07:56:44.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:44.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T07:56:45.513 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2677114065,v1:127.0.0.1:6827/2677114065] [v2:127.0.0.1:6828/2677114065,v1:127.0.0.1:6829/2677114065] exists,up 0b30b8ef-776f-4ca2-9e01-c3fd1a3ff04c 2024-09-16T07:56:45.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:56:45.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:56:45.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:56:45.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:56:45.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-16T07:56:45.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:56:45.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:56:45.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-16T07:56:45.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:56:45.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-16T07:56:45.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-16T07:56:45.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:56:45.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:56:45.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:56:45.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-16T07:56:45.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-16T07:56:45.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:56:45.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:56:45.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:56:45.516 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:56:45.516 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:56:45.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:56:45.519 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:56:45.519 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:56:45.519 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:56:45.519 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:56:45.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:56:45.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:56:45.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:56:45.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:56:45.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:56:45.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:56:45.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:56:45.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:56:45.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:56:45.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:56:45.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:56:45.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:56:45.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-16T07:56:45.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:56:45.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=687e347c-61f0-4ea8-9c29-a44b035d71cf 2024-09-16T07:56:45.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 687e347c-61f0-4ea8-9c29-a44b035d71cf' 2024-09-16T07:56:45.522 INFO:tasks.workunit.client.0.smithi162.stdout:add osd4 687e347c-61f0-4ea8-9c29-a44b035d71cf 2024-09-16T07:56:45.522 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:56:45.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC95OdmMA70IBAA9l5VLSsueRsYz5/36tIE5A== 2024-09-16T07:56:45.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC95OdmMA70IBAA9l5VLSsueRsYz5/36tIE5A=="}' 2024-09-16T07:56:45.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 687e347c-61f0-4ea8-9c29-a44b035d71cf -i td/test-erasure-eio/4/new.json 2024-09-16T07:56:46.626 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:56:46.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-16T07:56:46.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC95OdmMA70IBAA9l5VLSsueRsYz5/36tIE5A== --osd-uuid 687e347c-61f0-4ea8-9c29-a44b035d71cf 2024-09-16T07:56:46.680 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:56:46.678+0000 7fe691207880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:56:46.683 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:56:46.682+0000 7fe691207880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:56:46.684 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:56:46.682+0000 7fe691207880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:56:46.685 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:56:46.682+0000 7fe691207880 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-16T07:56:46.686 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:56:46.682+0000 7fe691207880 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-16T07:56:46.686 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:56:46.682+0000 7fe691207880 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-16T07:56:46.875 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:56:46.874+0000 7fe691207880 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-16T07:56:53.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-16T07:56:53.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:56:53.289 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd4 key to auth repository 2024-09-16T07:56:53.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-16T07:56:53.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:56:53.866 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.4 2024-09-16T07:56:53.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-16T07:56:53.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:56:53.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:56:53.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:56:53.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:56:53.904 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:56:53.902+0000 7f6d662a7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:56:53.919 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:56:53.918+0000 7f6d662a7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:56:53.920 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:56:53.918+0000 7f6d662a7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:56:54.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-16T07:56:54.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:56:54.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-16T07:56:54.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:56:54.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:56:54.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:54.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:56:54.292 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:56:54.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:54.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:56:54.472 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:56:54.470+0000 7f6d662a7880 -1 Falling back to public interface 2024-09-16T07:56:54.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:55.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:55.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:55.717 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:56:55.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:56:55.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:55.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:56:56.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:57.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:57.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:57.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:56:57.152 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:56:57.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:57.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:56:57.181 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:56:57.178+0000 7f6d662a7880 -1 osd.4 0 log_to_monitors true 2024-09-16T07:56:57.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:56:58.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:56:58.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:56:58.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:56:58.769 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:56:58.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:56:58.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:56:59.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:00.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:00.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:00.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:57:00.351 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:57:00.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:00.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:57:01.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:02.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:02.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:02.088 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:57:02.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:57:02.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:02.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:57:02.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:03.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:03.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:03.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:57:03.586 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:57:03.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:03.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:57:04.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:05.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:05.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:05.170 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:57:05.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:57:05.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:05.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:57:05.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:06.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:06.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:06.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:57:06.617 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:57:06.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:06.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:57:07.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:08.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:08.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:08.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:57:08.137 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:57:08.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:08.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:57:08.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:09.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:09.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:09.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:57:09.564 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:57:09.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:09.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:57:10.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:11.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:11.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:11.087 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:57:11.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:57:11.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:11.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:57:11.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:12.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:12.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:12.620 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:57:12.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:57:12.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:12.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:57:13.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:14.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:14.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:14.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:57:14.628 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:57:14.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:14.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:57:15.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:16.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:16.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:16.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:57:16.609 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:57:16.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:16.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:57:17.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:18.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:18.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:18.796 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:57:18.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:57:18.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:18.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:57:19.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:20.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:20.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:20.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:57:20.946 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:57:20.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:20.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:57:21.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:22.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:22.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:22.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:57:22.589 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:57:22.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:22.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:57:23.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:24.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:24.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:24.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T07:57:24.245 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-16T07:57:24.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:24.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T07:57:25.062 INFO:tasks.workunit.client.0.smithi162.stdout:osd.4 up in weight 1 up_from 39 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3669369078,v1:127.0.0.1:6835/3669369078] [v2:127.0.0.1:6836/3669369078,v1:127.0.0.1:6837/3669369078] exists,up 687e347c-61f0-4ea8-9c29-a44b035d71cf 2024-09-16T07:57:25.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:57:25.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:57:25.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:57:25.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:57:25.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-16T07:57:25.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:57:25.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:57:25.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-16T07:57:25.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:57:25.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-16T07:57:25.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-16T07:57:25.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:57:25.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:57:25.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:57:25.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-16T07:57:25.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-16T07:57:25.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:57:25.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:57:25.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:57:25.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:57:25.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:57:25.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:57:25.066 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:57:25.066 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:57:25.066 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:57:25.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:57:25.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:57:25.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:57:25.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:57:25.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:57:25.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:57:25.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:57:25.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:57:25.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:57:25.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:57:25.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:57:25.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:57:25.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:57:25.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-16T07:57:25.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:57:25.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=975f28fa-5f0b-4ce4-bedb-c7245ef1d9dc 2024-09-16T07:57:25.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 975f28fa-5f0b-4ce4-bedb-c7245ef1d9dc' 2024-09-16T07:57:25.071 INFO:tasks.workunit.client.0.smithi162.stdout:add osd5 975f28fa-5f0b-4ce4-bedb-c7245ef1d9dc 2024-09-16T07:57:25.071 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:57:25.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDl5OdmkO4FBhAAcJ9XqioL3sZ5DALR+u/gDg== 2024-09-16T07:57:25.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDl5OdmkO4FBhAAcJ9XqioL3sZ5DALR+u/gDg=="}' 2024-09-16T07:57:25.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 975f28fa-5f0b-4ce4-bedb-c7245ef1d9dc -i td/test-erasure-eio/5/new.json 2024-09-16T07:57:26.141 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:57:26.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-16T07:57:26.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDl5OdmkO4FBhAAcJ9XqioL3sZ5DALR+u/gDg== --osd-uuid 975f28fa-5f0b-4ce4-bedb-c7245ef1d9dc 2024-09-16T07:57:26.193 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:57:26.189+0000 7fe7a0c37880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:57:26.498 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:57:26.497+0000 7fe7a0c37880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:57:26.501 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:57:26.497+0000 7fe7a0c37880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:57:26.502 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:57:26.501+0000 7fe7a0c37880 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-16T07:57:26.502 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:57:26.501+0000 7fe7a0c37880 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-16T07:57:26.502 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:57:26.501+0000 7fe7a0c37880 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-16T07:57:26.610 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:57:26.609+0000 7fe7a0c37880 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-16T07:57:32.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-16T07:57:32.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:57:32.781 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd5 key to auth repository 2024-09-16T07:57:32.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-16T07:57:32.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:57:33.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-16T07:57:33.352 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.5 2024-09-16T07:57:33.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:57:33.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:57:33.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:57:33.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:57:33.391 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:57:33.389+0000 7fe52434b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:57:33.395 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:57:33.393+0000 7fe52434b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:57:33.397 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:57:33.393+0000 7fe52434b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:57:33.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-16T07:57:33.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:57:33.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-16T07:57:33.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:57:33.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:57:33.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:33.772 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:57:33.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:57:33.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:33.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:57:33.956 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:57:33.953+0000 7fe52434b880 -1 Falling back to public interface 2024-09-16T07:57:34.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:35.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:35.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:35.251 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:57:35.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:57:35.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:35.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:57:35.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:36.276 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:57:36.273+0000 7fe52434b880 -1 osd.5 0 log_to_monitors true 2024-09-16T07:57:36.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:36.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:36.688 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:57:36.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:57:36.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:36.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:57:37.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:38.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:38.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:38.215 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:57:38.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:57:38.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:38.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:57:38.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:39.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:39.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:39.749 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:57:39.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:57:39.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:39.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:57:40.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:41.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:41.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:41.376 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:57:41.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:57:41.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:41.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:57:41.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:42.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:42.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:42.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:57:42.909 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:57:42.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:57:42.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:43.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:44.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:44.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:44.496 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:57:44.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:57:44.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:44.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:57:44.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:45.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:45.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:45.945 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:57:45.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:57:45.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:45.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:57:46.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:47.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:47.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:47.438 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:57:47.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:57:47.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:47.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:57:48.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:49.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:49.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:49.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:57:49.116 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:57:49.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:49.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:57:49.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:50.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:50.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:50.603 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:57:50.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:57:50.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:50.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:57:51.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:52.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:52.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:52.532 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:57:52.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:57:52.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:52.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:57:53.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:54.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:54.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:54.602 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:57:54.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:57:54.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:54.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:57:55.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:56.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:56.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:56.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:57:56.491 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:57:56.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:56.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:57:57.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:57:58.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:57:58.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:57:58.360 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:57:58.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:57:58.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:57:58.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:57:59.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:58:00.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:58:00.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:58:00.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:58:00.355 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:58:00.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:58:00.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:58:01.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:58:02.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:58:02.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:58:02.392 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:58:02.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:58:02.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:58:02.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T07:58:02.828 INFO:tasks.workunit.client.0.smithi162.stdout:osd.5 up in weight 1 up_from 49 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/149975871,v1:127.0.0.1:6843/149975871] [v2:127.0.0.1:6844/149975871,v1:127.0.0.1:6845/149975871] exists,up 975f28fa-5f0b-4ce4-bedb-c7245ef1d9dc 2024-09-16T07:58:02.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:58:02.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:58:02.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:58:02.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:58:02.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-16T07:58:02.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:58:02.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:58:02.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-09-16T07:58:02.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:58:02.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-16T07:58:02.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-16T07:58:02.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:58:02.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:58:02.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:58:02.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-16T07:58:02.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-16T07:58:02.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:58:02.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:58:02.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:58:02.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:58:02.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:58:02.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:58:02.833 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:58:02.833 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:58:02.833 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:58:02.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:58:02.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:58:02.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:58:02.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:58:02.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:58:02.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:58:02.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:58:02.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:58:02.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:58:02.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:58:02.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:58:02.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:58:02.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:58:02.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-16T07:58:02.837 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:58:02.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4a863c29-734b-41de-830b-38fbb143965d 2024-09-16T07:58:02.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 4a863c29-734b-41de-830b-38fbb143965d' 2024-09-16T07:58:02.837 INFO:tasks.workunit.client.0.smithi162.stdout:add osd6 4a863c29-734b-41de-830b-38fbb143965d 2024-09-16T07:58:02.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:58:02.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAK5edmV71rMxAAtVNJBKRyWDi4RM6Aw3CqOg== 2024-09-16T07:58:02.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAK5edmV71rMxAAtVNJBKRyWDi4RM6Aw3CqOg=="}' 2024-09-16T07:58:02.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4a863c29-734b-41de-830b-38fbb143965d -i td/test-erasure-eio/6/new.json 2024-09-16T07:58:03.450 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:58:03.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-16T07:58:03.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAK5edmV71rMxAAtVNJBKRyWDi4RM6Aw3CqOg== --osd-uuid 4a863c29-734b-41de-830b-38fbb143965d 2024-09-16T07:58:03.509 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:58:03.504+0000 7f0e24de4880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:58:03.514 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:58:03.512+0000 7f0e24de4880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:58:03.516 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:58:03.512+0000 7f0e24de4880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:58:03.517 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:58:03.512+0000 7f0e24de4880 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-16T07:58:03.517 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:58:03.512+0000 7f0e24de4880 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-16T07:58:03.517 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:58:03.512+0000 7f0e24de4880 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-16T07:58:03.691 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:58:03.688+0000 7f0e24de4880 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-16T07:58:11.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-16T07:58:11.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:58:11.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-09-16T07:58:11.101 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd6 key to auth repository 2024-09-16T07:58:11.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:58:11.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-09-16T07:58:11.701 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.6 2024-09-16T07:58:11.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:58:11.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:58:11.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:58:11.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:58:11.740 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:58:11.736+0000 7f29cf305880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:58:11.775 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:58:11.772+0000 7f29cf305880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:58:11.777 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:58:11.772+0000 7f29cf305880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:58:12.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-09-16T07:58:12.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:58:12.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-09-16T07:58:12.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:58:12.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:58:12.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:58:12.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:58:12.155 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:58:12.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:58:12.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:58:12.339 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:58:12.336+0000 7f29cf305880 -1 Falling back to public interface 2024-09-16T07:58:12.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:58:13.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:58:13.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:58:13.596 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:58:13.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:58:13.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:58:13.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:58:14.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:58:14.997 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:58:14.992+0000 7f29cf305880 -1 osd.6 0 log_to_monitors true 2024-09-16T07:58:15.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:58:15.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:58:15.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:58:15.029 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:58:15.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:58:15.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:58:15.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:58:16.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:58:16.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:58:16.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:58:16.738 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:58:16.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:58:16.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:58:17.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:58:18.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:58:18.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:58:18.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:58:18.411 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:58:18.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:58:18.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:58:19.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:58:20.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:58:20.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:58:20.190 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:58:20.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:58:20.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:58:20.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:58:20.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:58:21.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:58:21.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:58:21.630 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:58:21.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:58:21.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:58:21.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:58:22.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:58:23.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:58:23.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:58:23.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:58:23.178 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:58:23.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:58:23.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:58:23.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:58:24.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:58:24.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:58:24.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:58:24.665 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:58:24.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:58:24.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:58:25.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:58:26.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:58:26.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:58:26.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:58:26.270 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:58:26.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:58:26.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:58:26.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:58:27.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:58:27.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:58:27.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:58:27.948 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:58:27.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:58:27.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:58:28.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:58:29.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:58:29.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:58:29.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:58:29.387 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:58:29.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:58:29.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:58:29.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:58:30.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:58:30.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:58:30.858 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:58:30.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:58:30.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:58:30.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:58:31.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:58:32.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:58:32.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:58:32.906 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:58:32.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:58:32.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:58:32.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:58:33.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:58:34.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:58:34.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:58:34.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:58:34.802 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:58:34.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:58:34.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:58:35.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:58:36.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:58:36.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:58:36.571 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:58:36.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:58:36.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:58:36.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:58:37.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:58:38.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:58:38.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:58:38.538 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:58:38.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:58:38.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:58:38.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:58:39.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:58:40.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:58:40.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:58:40.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T07:58:40.547 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T07:58:40.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:58:40.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:58:41.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:58:42.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:58:42.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:58:42.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T07:58:42.138 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-16T07:58:42.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:58:42.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T07:58:42.862 INFO:tasks.workunit.client.0.smithi162.stdout:osd.6 up in weight 1 up_from 60 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/4006284696,v1:127.0.0.1:6851/4006284696] [v2:127.0.0.1:6852/4006284696,v1:127.0.0.1:6853/4006284696] exists,up 4a863c29-734b-41de-830b-38fbb143965d 2024-09-16T07:58:42.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:58:42.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:58:42.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:58:42.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-16T07:58:42.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T07:58:42.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T07:58:42.864 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:58:42.864 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:58:42.865 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:58:42.865 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.0.asok 2024-09-16T07:58:42.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-16T07:58:42.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.0.asok log flush 2024-09-16T07:58:42.962 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:58:42.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-16T07:58:42.982 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:54:26.157+0000 7ffb7e778880 0 load: jerasure load: lrc 2024-09-16T07:58:42.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:486: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS='--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:58:42.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:488: TEST_ec_recovery_multiple_objects_eio: local poolname=pool-jerasure 2024-09-16T07:58:42.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:489: TEST_ec_recovery_multiple_objects_eio: create_erasure_coded_pool pool-jerasure 3 2 2024-09-16T07:58:42.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T07:58:42.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-16T07:58:42.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-16T07:58:42.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-16T07:58:42.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-16T07:58:42.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-16T07:58:42.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-16T07:58:43.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-16T07:58:43.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-16T07:58:44.845 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-16T07:58:44.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T07:58:45.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-16T07:58:45.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T07:58:45.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T07:58:45.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T07:58:45.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T07:58:45.857 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T07:58:45.858 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T07:58:45.858 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T07:58:45.858 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T07:58:45.858 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T07:58:45.858 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T07:58:46.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T07:58:46.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T07:58:46.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T07:58:46.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T07:58:46.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T07:58:46.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T07:58:46.520 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:58:46.521 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:58:46.521 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:58:46.521 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:58:46.521 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:58:46.521 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-16T07:58:46.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T07:58:46.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:58:46.522 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T07:58:46.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803834 2024-09-16T07:58:46.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803834 2024-09-16T07:58:46.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803834' 2024-09-16T07:58:46.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:58:46.808 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T07:58:47.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607600 2024-09-16T07:58:47.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607600 2024-09-16T07:58:47.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803834 1-51539607600' 2024-09-16T07:58:47.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:58:47.045 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T07:58:47.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=77309411371 2024-09-16T07:58:47.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 77309411371 2024-09-16T07:58:47.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803834 1-51539607600 2-77309411371' 2024-09-16T07:58:47.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:58:47.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T07:58:47.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=120259084323 2024-09-16T07:58:47.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 120259084323 2024-09-16T07:58:47.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803834 1-51539607600 2-77309411371 3-120259084323' 2024-09-16T07:58:47.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:58:47.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T07:58:47.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=167503724574 2024-09-16T07:58:47.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 167503724574 2024-09-16T07:58:47.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803834 1-51539607600 2-77309411371 3-120259084323 4-167503724574' 2024-09-16T07:58:47.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:58:47.499 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T07:58:47.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=210453397527 2024-09-16T07:58:47.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 210453397527 2024-09-16T07:58:47.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803834 1-51539607600 2-77309411371 3-120259084323 4-167503724574 5-210453397527' 2024-09-16T07:58:47.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:58:47.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-16T07:58:47.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=257698037778 2024-09-16T07:58:47.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 257698037778 2024-09-16T07:58:47.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803834 1-51539607600 2-77309411371 3-120259084323 4-167503724574 5-210453397527 6-257698037778' 2024-09-16T07:58:47.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:58:47.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803834 2024-09-16T07:58:47.819 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:58:47.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T07:58:47.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803834 2024-09-16T07:58:47.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:58:47.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803834 2024-09-16T07:58:47.824 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803834 2024-09-16T07:58:47.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803834' 2024-09-16T07:58:47.825 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:58:48.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803832 -lt 25769803834 2024-09-16T07:58:48.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T07:58:49.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T07:58:49.251 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:58:49.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803834 -lt 25769803834 2024-09-16T07:58:49.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:58:49.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607600 2024-09-16T07:58:49.693 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:58:49.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T07:58:49.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607600 2024-09-16T07:58:49.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:58:49.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607600 2024-09-16T07:58:49.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607600' 2024-09-16T07:58:49.699 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 51539607600 2024-09-16T07:58:49.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T07:58:50.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607600 -lt 51539607600 2024-09-16T07:58:50.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:58:50.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-77309411371 2024-09-16T07:58:50.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:58:50.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T07:58:50.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-77309411371 2024-09-16T07:58:50.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:58:50.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=77309411371 2024-09-16T07:58:50.210 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 77309411371 2024-09-16T07:58:50.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 77309411371' 2024-09-16T07:58:50.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T07:58:50.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 77309411371 -lt 77309411371 2024-09-16T07:58:50.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:58:50.607 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-120259084323 2024-09-16T07:58:50.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:58:50.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T07:58:50.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-120259084323 2024-09-16T07:58:50.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:58:50.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=120259084323 2024-09-16T07:58:50.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 120259084323' 2024-09-16T07:58:50.613 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 120259084323 2024-09-16T07:58:50.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T07:58:51.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 120259084324 -lt 120259084323 2024-09-16T07:58:51.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:58:51.057 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:58:51.058 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-167503724574 2024-09-16T07:58:51.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T07:58:51.060 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:58:51.060 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-167503724574 2024-09-16T07:58:51.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=167503724574 2024-09-16T07:58:51.062 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 167503724574 2024-09-16T07:58:51.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 167503724574' 2024-09-16T07:58:51.062 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T07:58:51.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 167503724575 -lt 167503724574 2024-09-16T07:58:51.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:58:51.547 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-210453397527 2024-09-16T07:58:51.547 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:58:51.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T07:58:51.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-210453397527 2024-09-16T07:58:51.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:58:51.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=210453397527 2024-09-16T07:58:51.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 210453397527' 2024-09-16T07:58:51.551 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 210453397527 2024-09-16T07:58:51.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T07:58:52.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 210453397528 -lt 210453397527 2024-09-16T07:58:52.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:58:52.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 6-257698037778 2024-09-16T07:58:52.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:58:52.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=6 2024-09-16T07:58:52.017 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 6-257698037778 2024-09-16T07:58:52.018 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:58:52.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=257698037778 2024-09-16T07:58:52.020 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 257698037778 2024-09-16T07:58:52.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.6 seq 257698037778' 2024-09-16T07:58:52.020 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-16T07:58:52.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 257698037778 -lt 257698037778 2024-09-16T07:58:52.424 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T07:58:52.424 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:58:52.424 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:58:53.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T07:58:53.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T07:58:53.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T07:58:53.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T07:58:53.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T07:58:53.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T07:58:53.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T07:58:53.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T07:58:53.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T07:58:53.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T07:58:53.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:58:53.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:58:53.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T07:58:53.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T07:58:53.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T07:58:53.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:491: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test1 2024-09-16T07:58:53.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-16T07:58:53.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-16T07:58:53.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2024-09-16T07:58:53.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:58:53.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-16T07:58:53.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:58:53.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-16T07:58:53.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:58:53.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-16T07:58:53.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:58:53.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-16T07:58:53.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2024-09-16T07:58:54.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:492: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test2 2024-09-16T07:58:54.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-16T07:58:54.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-16T07:58:54.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2024-09-16T07:58:54.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:58:54.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-16T07:58:54.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:58:54.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-16T07:58:54.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:58:54.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-16T07:58:54.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:58:54.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-16T07:58:54.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2024-09-16T07:58:54.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:493: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test3 2024-09-16T07:58:54.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-16T07:58:54.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-16T07:58:54.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2024-09-16T07:58:54.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:58:54.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-16T07:58:54.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:58:54.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-16T07:58:54.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:58:54.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-16T07:58:54.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T07:58:54.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-16T07:58:54.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2024-09-16T07:58:54.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:496: TEST_ec_recovery_multiple_objects_eio: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-09-16T07:58:54.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=ec 2024-09-16T07:58:54.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T07:58:54.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T07:58:54.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T07:58:54.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-jerasure 2024-09-16T07:58:54.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T07:58:54.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=myobject 2024-09-16T07:58:54.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T07:58:54.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/test-erasure-eio 2024-09-16T07:58:54.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T07:58:54.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T07:58:54.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T07:58:54.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T07:58:54.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-jerasure myobject 2024-09-16T07:58:54.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T07:58:54.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-16T07:58:54.823 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-16T07:58:54.823 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T07:58:55.234 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T07:58:55.234 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:58:55.234 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:58:55.234 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T07:58:55.234 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-16T07:58:55.234 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2024-09-16T07:58:55.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T07:58:55.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T07:58:55.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' ec '!=' ec ']' 2024-09-16T07:58:55.235 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/test-erasure-eio/3/type 2024-09-16T07:58:55.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T07:58:55.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T07:58:55.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T07:58:55.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T07:58:55.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T07:58:55.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T07:58:55.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T07:58:55.239 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T07:58:55.239 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T07:58:55.239 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T07:58:55.240 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:58:55.240 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:58:55.240 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:58:55.241 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.3.asok 2024-09-16T07:58:55.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79228/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T07:58:55.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T07:58:55.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T07:58:55.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T07:58:55.344 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T07:58:55.344 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T07:58:55.344 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T07:58:55.344 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:58:55.344 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:58:55.345 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:58:55.345 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.3.asok 2024-09-16T07:58:55.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T07:58:55.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-09-16T07:58:55.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:497: TEST_ec_recovery_multiple_objects_eio: ceph osd out 0 2024-09-16T07:58:56.225 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 is already out. 2024-09-16T07:58:56.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:500: TEST_ec_recovery_multiple_objects_eio: wait_for_clean 2024-09-16T07:58:56.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T07:58:56.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T07:58:56.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T07:58:56.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T07:58:56.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T07:58:56.241 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T07:58:56.241 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T07:58:56.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T07:58:56.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T07:58:56.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T07:58:56.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T07:58:56.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T07:58:56.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T07:58:56.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T07:58:56.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T07:58:56.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T07:58:56.980 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T07:58:56.980 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T07:58:56.980 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T07:58:56.980 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T07:58:56.980 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T07:58:56.981 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-16T07:58:56.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T07:58:56.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:58:56.981 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T07:58:57.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803838 2024-09-16T07:58:57.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803838 2024-09-16T07:58:57.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803838' 2024-09-16T07:58:57.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:58:57.128 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T07:58:57.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607604 2024-09-16T07:58:57.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607604 2024-09-16T07:58:57.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803838 1-51539607604' 2024-09-16T07:58:57.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:58:57.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T07:58:57.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=77309411375 2024-09-16T07:58:57.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 77309411375 2024-09-16T07:58:57.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803838 1-51539607604 2-77309411375' 2024-09-16T07:58:57.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:58:57.506 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T07:58:57.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=120259084327 2024-09-16T07:58:57.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 120259084327 2024-09-16T07:58:57.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803838 1-51539607604 2-77309411375 3-120259084327' 2024-09-16T07:58:57.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:58:57.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T07:58:57.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=167503724578 2024-09-16T07:58:57.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 167503724578 2024-09-16T07:58:57.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803838 1-51539607604 2-77309411375 3-120259084327 4-167503724578' 2024-09-16T07:58:57.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:58:57.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T07:58:58.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=210453397532 2024-09-16T07:58:58.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 210453397532 2024-09-16T07:58:58.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803838 1-51539607604 2-77309411375 3-120259084327 4-167503724578 5-210453397532' 2024-09-16T07:58:58.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T07:58:58.140 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-16T07:58:58.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=257698037782 2024-09-16T07:58:58.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 257698037782 2024-09-16T07:58:58.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803838 1-51539607604 2-77309411375 3-120259084327 4-167503724578 5-210453397532 6-257698037782' 2024-09-16T07:58:58.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:58:58.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:58:58.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803838 2024-09-16T07:58:58.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T07:58:58.312 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803838 2024-09-16T07:58:58.312 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:58:58.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803838 2024-09-16T07:58:58.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803838' 2024-09-16T07:58:58.314 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803838 2024-09-16T07:58:58.315 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:58:58.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 0 -lt 25769803838 2024-09-16T07:58:58.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T07:58:59.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T07:58:59.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T07:59:00.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803838 -lt 25769803838 2024-09-16T07:59:00.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:59:00.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607604 2024-09-16T07:59:00.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:59:00.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T07:59:00.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607604 2024-09-16T07:59:00.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:59:00.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607604 2024-09-16T07:59:00.187 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 51539607604 2024-09-16T07:59:00.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607604' 2024-09-16T07:59:00.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T07:59:00.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607604 -lt 51539607604 2024-09-16T07:59:00.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:59:00.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-77309411375 2024-09-16T07:59:00.635 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:59:00.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T07:59:00.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-77309411375 2024-09-16T07:59:00.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:59:00.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=77309411375 2024-09-16T07:59:00.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 77309411375' 2024-09-16T07:59:00.639 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 77309411375 2024-09-16T07:59:00.640 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T07:59:01.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 77309411375 -lt 77309411375 2024-09-16T07:59:01.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:59:01.023 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-120259084327 2024-09-16T07:59:01.023 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:59:01.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T07:59:01.027 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-120259084327 2024-09-16T07:59:01.027 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:59:01.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=120259084327 2024-09-16T07:59:01.029 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 120259084327 2024-09-16T07:59:01.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 120259084327' 2024-09-16T07:59:01.029 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T07:59:01.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 120259084328 -lt 120259084327 2024-09-16T07:59:01.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:59:01.546 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-167503724578 2024-09-16T07:59:01.546 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:59:01.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T07:59:01.550 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-167503724578 2024-09-16T07:59:01.550 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:59:01.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=167503724578 2024-09-16T07:59:01.552 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 167503724578 2024-09-16T07:59:01.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 167503724578' 2024-09-16T07:59:01.553 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T07:59:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 167503724579 -lt 167503724578 2024-09-16T07:59:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:59:01.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-210453397532 2024-09-16T07:59:01.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:59:01.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T07:59:01.986 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-210453397532 2024-09-16T07:59:01.986 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:59:01.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=210453397532 2024-09-16T07:59:01.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 210453397532' 2024-09-16T07:59:01.989 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 210453397532 2024-09-16T07:59:01.989 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T07:59:02.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 210453397532 -lt 210453397532 2024-09-16T07:59:02.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T07:59:02.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 6-257698037782 2024-09-16T07:59:02.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T07:59:02.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=6 2024-09-16T07:59:02.530 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 6-257698037782 2024-09-16T07:59:02.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T07:59:02.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=257698037782 2024-09-16T07:59:02.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.6 seq 257698037782' 2024-09-16T07:59:02.534 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 257698037782 2024-09-16T07:59:02.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-16T07:59:02.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 257698037783 -lt 257698037782 2024-09-16T07:59:02.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T07:59:02.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:59:02.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:59:03.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T07:59:03.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T07:59:03.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T07:59:03.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T07:59:03.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T07:59:03.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T07:59:03.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T07:59:03.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T07:59:03.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T07:59:03.949 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T07:59:03.949 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T07:59:03.949 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T07:59:04.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T07:59:04.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T07:59:04.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T07:59:04.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:502: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test1 2024-09-16T07:59:04.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T07:59:04.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T07:59:04.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2024-09-16T07:59:04.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-16T07:59:04.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-16T07:59:04.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2024-09-16T07:59:04.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-16T07:59:04.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-16T07:59:04.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:503: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test2 2024-09-16T07:59:04.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T07:59:04.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T07:59:04.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2024-09-16T07:59:04.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-16T07:59:04.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-16T07:59:04.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2024-09-16T07:59:04.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-16T07:59:04.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-16T07:59:04.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:504: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test3 2024-09-16T07:59:04.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T07:59:04.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T07:59:04.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2024-09-16T07:59:04.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-16T07:59:04.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-16T07:59:04.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2024-09-16T07:59:04.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-16T07:59:04.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-16T07:59:04.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:506: TEST_ec_recovery_multiple_objects_eio: delete_erasure_coded_pool pool-jerasure 2024-09-16T07:59:04.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T07:59:04.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-16T07:59:05.434 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-16T07:59:05.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-16T07:59:06.128 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-16T07:59:06.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-16T07:59:06.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T07:59:06.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:59:06.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T07:59:06.141 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:59:06.141 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:59:06.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:59:06.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:59:06.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:59:06.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:59:06.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:59:06.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:59:06.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:59:06.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:59:06.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:59:06.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:59:06.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:59:06.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:59:06.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:59:06.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:59:06.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:59:06.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:59:06.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T07:59:06.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:59:06.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:59:06.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:59:06.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T07:59:06.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:59:06.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:59:06.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-16T07:59:06.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-16T07:59:06.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-16T07:59:06.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-16T07:59:06.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T07:59:06.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T07:59:06.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T07:59:06.522 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T07:59:06.522 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T07:59:06.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T07:59:06.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T07:59:06.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T07:59:06.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T07:59:06.526 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T07:59:06.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T07:59:06.529 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T07:59:06.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T07:59:06.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T07:59:06.532 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T07:59:06.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:59:06.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T07:59:06.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T07:59:06.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:59:06.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:59:06.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T07:59:06.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T07:59:06.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T07:59:06.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:59:06.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:59:06.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T07:59:06.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T07:59:06.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T07:59:06.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-16T07:59:06.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T07:59:06.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:59:06.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:59:06.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79228 2024-09-16T07:59:06.548 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T07:59:06.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T07:59:06.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T07:59:06.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-16T07:59:06.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-16T07:59:06.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-16T07:59:06.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T07:59:06.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T07:59:06.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T07:59:06.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-16T07:59:06.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-16T07:59:07.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T07:59:07.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:59:07.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:59:07.016 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:59:07.016 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:59:07.016 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:59:07.017 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:59:07.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T07:59:07.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T07:59:07.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T07:59:07.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:59:07.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:59:07.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T07:59:07.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T07:59:07.240 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:59:07.240 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:59:07.240 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:59:07.241 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:59:07.241 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:59:07.241 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:59:07.242 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T07:59:07.243 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:59:07.243 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get fsid 2024-09-16T07:59:07.344 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T07:59:07.344 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T07:59:07.344 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T07:59:07.345 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T07:59:07.345 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T07:59:07.345 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T07:59:07.346 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:59:07.346 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:59:07.346 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:59:07.346 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:59:07.346 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:59:07.346 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T07:59:07.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T07:59:07.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get mon_host 2024-09-16T07:59:07.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-16T07:59:07.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-16T07:59:07.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T07:59:07.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T07:59:07.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T07:59:07.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-16T07:59:07.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T07:59:08.356 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T07:59:08.357 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:59:08.357 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:59:08.357 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:59:08.357 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:59:08.357 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:59:08.358 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:59:08.358 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:59:08.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T07:59:08.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-16T07:59:08.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T07:59:08.778 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-16T07:59:08.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T07:59:09.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-16T07:59:09.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T07:59:09.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T07:59:09.790 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T07:59:09.791 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:59:09.791 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:59:09.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T07:59:09.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-16T07:59:09.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-mon.a.asok log flush 2024-09-16T07:59:09.875 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T07:59:09.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-16T07:59:09.886 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:59:07.063+0000 7f8ad5e43d40 0 load: jerasure load: lrc 2024-09-16T07:59:09.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_unfound td/test-erasure-eio 2024-09-16T07:59:09.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:609: TEST_ec_recovery_unfound: local dir=td/test-erasure-eio 2024-09-16T07:59:09.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:610: TEST_ec_recovery_unfound: local objname=myobject 2024-09-16T07:59:09.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_recovery_unfound: local lastobj=100 2024-09-16T07:59:09.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_recovery_unfound: local testobj=obj75 2024-09-16T07:59:09.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:615: TEST_ec_recovery_unfound: ORIG_ARGS='--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T07:59:09.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:616: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-16T07:59:09.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-16T07:59:09.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_recovery_unfound: setup_osds 5 2024-09-16T07:59:09.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2024-09-16T07:59:09.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-16T07:59:09.890 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2024-09-16T07:59:09.890 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2024-09-16T07:59:09.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:59:09.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-16T07:59:09.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:59:09.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:59:09.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T07:59:09.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:59:09.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-16T07:59:09.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-16T07:59:09.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:59:09.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:59:09.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:59:09.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-16T07:59:09.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-16T07:59:09.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:59:09.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:59:09.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:59:09.894 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:59:09.894 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:59:09.894 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:59:09.895 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:59:09.896 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:59:09.896 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:59:09.896 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:59:09.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:59:09.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:59:09.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:59:09.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:59:09.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:59:09.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:59:09.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:59:09.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:59:09.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:59:09.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:59:09.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:59:09.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:59:09.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-16T07:59:09.897 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:59:09.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=71a47bfc-acf4-4a51-864c-50eb3c43f2f2 2024-09-16T07:59:09.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 71a47bfc-acf4-4a51-864c-50eb3c43f2f2' 2024-09-16T07:59:09.898 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 71a47bfc-acf4-4a51-864c-50eb3c43f2f2 2024-09-16T07:59:09.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:59:09.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBN5edmB0keNxAAwy/5Jvz3giaXD8rBZwP4xw== 2024-09-16T07:59:09.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBN5edmB0keNxAAwy/5Jvz3giaXD8rBZwP4xw=="}' 2024-09-16T07:59:09.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 71a47bfc-acf4-4a51-864c-50eb3c43f2f2 -i td/test-erasure-eio/0/new.json 2024-09-16T07:59:10.292 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:59:10.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-16T07:59:10.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBN5edmB0keNxAAwy/5Jvz3giaXD8rBZwP4xw== --osd-uuid 71a47bfc-acf4-4a51-864c-50eb3c43f2f2 2024-09-16T07:59:10.353 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:59:10.347+0000 7fd1baeeb880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:59:10.400 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:59:10.395+0000 7fd1baeeb880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:59:10.402 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:59:10.399+0000 7fd1baeeb880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:59:10.403 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:59:10.399+0000 7fd1baeeb880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T07:59:10.403 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:59:10.399+0000 7fd1baeeb880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T07:59:10.403 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:59:10.399+0000 7fd1baeeb880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T07:59:10.492 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:59:10.487+0000 7fd1baeeb880 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-16T07:59:16.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-16T07:59:16.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:59:16.173 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T07:59:16.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T07:59:16.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:59:16.743 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T07:59:16.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T07:59:16.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:59:16.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:59:16.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:59:16.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:59:16.785 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:59:16.779+0000 7f5d0e66e880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:59:16.810 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:59:16.807+0000 7f5d0e66e880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:59:16.812 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:59:16.807+0000 7f5d0e66e880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:59:17.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T07:59:17.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:59:17.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T07:59:17.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:59:17.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:59:17.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:59:17.203 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:59:17.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:59:17.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:59:17.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:59:17.376 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:59:17.371+0000 7f5d0e66e880 -1 Falling back to public interface 2024-09-16T07:59:17.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:59:18.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:59:18.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:59:18.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:59:18.662 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:59:18.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:59:18.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:59:19.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:59:19.755 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:59:19.751+0000 7f5d0e66e880 -1 osd.0 0 log_to_monitors true 2024-09-16T07:59:20.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:59:20.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:59:20.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:59:20.095 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:59:20.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:59:20.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:59:20.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:59:21.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:59:21.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:59:21.653 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:59:21.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:59:21.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:59:21.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:59:22.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:59:23.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:59:23.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:59:23.136 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:59:23.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:59:23.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:59:23.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:59:23.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:59:24.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:59:24.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:59:24.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T07:59:24.629 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T07:59:24.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:59:24.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:59:25.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:59:26.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:59:26.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:59:26.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T07:59:26.137 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T07:59:26.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:59:26.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:59:26.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:59:27.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:59:27.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:59:27.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T07:59:27.639 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T07:59:27.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:59:27.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:59:28.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:59:29.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:59:29.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:59:29.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T07:59:29.100 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T07:59:29.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:59:29.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:59:29.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:59:30.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:59:30.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:59:30.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T07:59:30.662 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T07:59:30.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:59:30.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:59:31.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:59:32.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:59:32.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:59:32.130 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T07:59:32.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T07:59:32.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:59:32.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:59:33.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:59:34.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:59:34.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:59:34.008 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T07:59:34.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T07:59:34.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:59:34.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:59:34.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:59:35.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:59:35.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:59:35.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T07:59:35.765 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T07:59:35.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:59:35.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:59:36.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:59:37.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:59:37.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:59:37.764 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T07:59:37.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T07:59:37.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:59:37.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:59:38.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:59:39.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:59:39.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:59:39.412 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T07:59:39.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T07:59:39.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:59:39.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:59:40.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:59:41.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:59:41.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:59:41.699 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T07:59:41.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T07:59:41.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:59:41.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:59:42.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:59:43.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:59:43.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:59:43.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T07:59:43.607 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T07:59:43.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:59:43.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T07:59:44.212 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1396991985,v1:127.0.0.1:6803/1396991985] [v2:127.0.0.1:6804/1396991985,v1:127.0.0.1:6805/1396991985] exists,up 71a47bfc-acf4-4a51-864c-50eb3c43f2f2 2024-09-16T07:59:44.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T07:59:44.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T07:59:44.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T07:59:44.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T07:59:44.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-16T07:59:44.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T07:59:44.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T07:59:44.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T07:59:44.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T07:59:44.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-16T07:59:44.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-16T07:59:44.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T07:59:44.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T07:59:44.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T07:59:44.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-16T07:59:44.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-16T07:59:44.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T07:59:44.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T07:59:44.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T07:59:44.224 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T07:59:44.224 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T07:59:44.224 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T07:59:44.224 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T07:59:44.224 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T07:59:44.225 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T07:59:44.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:59:44.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T07:59:44.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T07:59:44.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T07:59:44.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T07:59:44.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T07:59:44.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T07:59:44.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T07:59:44.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T07:59:44.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T07:59:44.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T07:59:44.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T07:59:44.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T07:59:44.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-16T07:59:44.228 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T07:59:44.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2297600a-e9be-44ff-ad7c-855ba9757a7b 2024-09-16T07:59:44.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2297600a-e9be-44ff-ad7c-855ba9757a7b' 2024-09-16T07:59:44.229 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 2297600a-e9be-44ff-ad7c-855ba9757a7b 2024-09-16T07:59:44.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T07:59:44.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBw5edm94u9DhAA0S6sMVurguW++Z4AUzCq+A== 2024-09-16T07:59:44.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBw5edm94u9DhAA0S6sMVurguW++Z4AUzCq+A=="}' 2024-09-16T07:59:44.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2297600a-e9be-44ff-ad7c-855ba9757a7b -i td/test-erasure-eio/1/new.json 2024-09-16T07:59:45.129 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:59:45.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-16T07:59:45.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBw5edm94u9DhAA0S6sMVurguW++Z4AUzCq+A== --osd-uuid 2297600a-e9be-44ff-ad7c-855ba9757a7b 2024-09-16T07:59:45.195 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:59:45.190+0000 7fc1ad309880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:59:45.200 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:59:45.194+0000 7fc1ad309880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:59:45.202 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:59:45.198+0000 7fc1ad309880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:59:45.203 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:59:45.198+0000 7fc1ad309880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T07:59:45.203 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:59:45.198+0000 7fc1ad309880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T07:59:45.203 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:59:45.198+0000 7fc1ad309880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T07:59:45.285 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:59:45.282+0000 7fc1ad309880 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-16T07:59:51.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-16T07:59:51.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T07:59:51.764 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-16T07:59:51.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T07:59:51.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T07:59:52.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T07:59:52.329 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-16T07:59:52.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T07:59:52.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T07:59:52.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T07:59:52.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T07:59:52.370 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:59:52.366+0000 7f634a65d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:59:52.429 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:59:52.426+0000 7f634a65d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:59:52.431 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:59:52.426+0000 7f634a65d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T07:59:52.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T07:59:52.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T07:59:52.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T07:59:52.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T07:59:52.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T07:59:52.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:59:52.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T07:59:52.815 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T07:59:52.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:59:52.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:59:53.004 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:59:52.998+0000 7f634a65d880 -1 Falling back to public interface 2024-09-16T07:59:53.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:59:54.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:59:54.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:59:54.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T07:59:54.316 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T07:59:54.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:59:54.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:59:54.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:59:55.291 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T07:59:55.290+0000 7f634a65d880 -1 osd.1 0 log_to_monitors true 2024-09-16T07:59:55.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:59:55.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:59:55.814 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T07:59:55.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T07:59:55.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:59:55.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:59:56.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:59:57.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:59:57.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:59:57.575 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T07:59:57.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T07:59:57.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:59:57.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:59:58.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T07:59:59.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T07:59:59.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T07:59:59.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T07:59:59.223 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T07:59:59.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T07:59:59.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T07:59:59.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:00.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:00.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:00.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:00:00.688 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:00:00.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:00.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:00:01.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:02.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:02.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:02.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:00:02.236 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:00:02.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:02.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:00:02.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:03.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:03.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:03.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:00:03.796 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:00:03.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:03.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:00:04.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:05.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:05.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:05.316 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:00:05.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:00:05.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:05.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:00:05.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:06.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:06.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:06.777 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:00:06.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:00:06.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:06.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:00:07.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:08.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:08.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:08.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:00:08.317 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:00:08.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:08.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:00:09.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:10.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:10.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:10.081 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:00:10.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:00:10.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:10.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:00:10.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:11.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:11.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:11.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:00:11.917 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:00:11.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:11.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:00:12.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:13.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:13.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:13.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:00:13.959 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:00:13.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:13.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:00:14.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:15.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:15.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:15.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:00:15.732 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:00:15.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:15.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:00:16.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:17.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:17.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:17.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:00:17.821 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:00:17.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:17.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:00:18.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:19.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:19.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:19.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:00:19.610 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:00:19.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:19.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:00:20.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:21.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:21.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:21.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:00:21.373 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:00:21.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:21.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:00:22.033 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3397346955,v1:127.0.0.1:6811/3397346955] [v2:127.0.0.1:6812/3397346955,v1:127.0.0.1:6813/3397346955] exists,up 2297600a-e9be-44ff-ad7c-855ba9757a7b 2024-09-16T08:00:22.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:00:22.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:00:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:00:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:00:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-16T08:00:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:00:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:00:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T08:00:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:00:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-16T08:00:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-16T08:00:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:00:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:00:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:00:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-16T08:00:22.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-16T08:00:22.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:00:22.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:00:22.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:00:22.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:00:22.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:00:22.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:00:22.039 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:00:22.039 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:00:22.039 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:00:22.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:00:22.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:00:22.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:00:22.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:00:22.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:00:22.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:00:22.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:00:22.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:00:22.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:00:22.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:00:22.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:00:22.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:00:22.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:00:22.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-16T08:00:22.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:00:22.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=90367fc6-7334-4796-9ee0-844712945b4c 2024-09-16T08:00:22.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 90367fc6-7334-4796-9ee0-844712945b4c' 2024-09-16T08:00:22.042 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 90367fc6-7334-4796-9ee0-844712945b4c 2024-09-16T08:00:22.043 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:00:22.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCW5edmOb5rBBAALSMhEZ1UCkAyRLfAiazSMw== 2024-09-16T08:00:22.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCW5edmOb5rBBAALSMhEZ1UCkAyRLfAiazSMw=="}' 2024-09-16T08:00:22.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 90367fc6-7334-4796-9ee0-844712945b4c -i td/test-erasure-eio/2/new.json 2024-09-16T08:00:22.626 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:00:22.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-16T08:00:22.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCW5edmOb5rBBAALSMhEZ1UCkAyRLfAiazSMw== --osd-uuid 90367fc6-7334-4796-9ee0-844712945b4c 2024-09-16T08:00:22.689 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:00:22.686+0000 7f42c4ca7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:00:22.708 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:00:22.706+0000 7f42c4ca7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:00:22.710 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:00:22.710+0000 7f42c4ca7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:00:22.711 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:00:22.710+0000 7f42c4ca7880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:00:22.711 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:00:22.710+0000 7f42c4ca7880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:00:22.711 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:00:22.710+0000 7f42c4ca7880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:00:22.840 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:00:22.838+0000 7f42c4ca7880 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-16T08:00:29.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-16T08:00:29.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:00:29.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T08:00:29.409 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-16T08:00:29.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:00:29.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T08:00:29.997 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-16T08:00:29.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:00:29.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:00:29.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:00:29.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:00:30.037 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:00:30.033+0000 7f47c181a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:00:30.040 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:00:30.037+0000 7f47c181a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:00:30.042 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:00:30.041+0000 7f47c181a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:00:30.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T08:00:30.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:00:30.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T08:00:30.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:00:30.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:00:30.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:30.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:00:30.436 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:00:30.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:30.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:00:30.604 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:00:30.601+0000 7f47c181a880 -1 Falling back to public interface 2024-09-16T08:00:30.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:31.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:31.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:31.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:00:31.872 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:00:31.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:31.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:00:32.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:33.221 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:00:33.217+0000 7f47c181a880 -1 osd.2 0 log_to_monitors true 2024-09-16T08:00:33.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:33.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:33.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:00:33.310 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:00:33.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:33.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:00:33.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:34.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:34.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:34.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:00:34.981 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:00:34.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:34.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:00:35.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:36.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:36.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:36.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:00:36.534 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:00:36.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:36.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:00:37.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:38.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:38.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:38.153 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:00:38.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:00:38.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:38.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:00:38.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:39.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:39.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:39.598 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:00:39.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:00:39.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:39.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:00:40.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:41.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:41.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:41.177 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:00:41.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:00:41.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:41.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:00:41.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:42.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:42.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:42.645 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:00:42.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:00:42.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:42.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:00:43.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:44.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:44.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:44.186 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:00:44.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:00:44.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:44.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:00:44.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:45.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:45.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:45.843 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:00:45.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:00:45.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:45.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:00:46.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:47.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:47.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:47.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:00:47.470 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:00:47.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:47.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:00:48.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:49.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:49.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:49.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:00:49.185 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:00:49.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:49.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:00:49.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:50.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:50.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:50.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:00:50.949 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:00:50.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:50.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:00:51.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:52.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:52.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:52.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:00:52.396 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:00:52.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:52.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:00:52.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:53.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:53.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:53.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:00:53.867 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:00:53.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:53.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:00:54.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:55.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:55.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:55.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:00:55.718 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:00:55.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:55.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:00:56.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:57.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:57.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:57.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:00:57.667 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:00:57.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:57.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:00:58.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:00:59.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:00:59.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:00:59.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T08:00:59.235 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-16T08:00:59.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:00:59.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:00:59.655 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2446207194,v1:127.0.0.1:6819/2446207194] [v2:127.0.0.1:6820/2446207194,v1:127.0.0.1:6821/2446207194] exists,up 90367fc6-7334-4796-9ee0-844712945b4c 2024-09-16T08:00:59.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:00:59.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:00:59.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:00:59.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:00:59.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-16T08:00:59.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:00:59.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:00:59.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T08:00:59.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:00:59.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-16T08:00:59.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-16T08:00:59.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:00:59.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:00:59.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:00:59.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-16T08:00:59.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-16T08:00:59.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:00:59.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:00:59.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:00:59.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:00:59.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:00:59.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:00:59.660 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:00:59.660 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:00:59.660 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:00:59.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:00:59.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:00:59.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:00:59.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:00:59.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:00:59.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:00:59.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:00:59.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:00:59.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:00:59.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:00:59.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:00:59.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:00:59.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:00:59.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-16T08:00:59.663 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:00:59.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e4868e0a-f7c6-4d86-b7a6-30bb0540a39f 2024-09-16T08:00:59.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 e4868e0a-f7c6-4d86-b7a6-30bb0540a39f' 2024-09-16T08:00:59.664 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 e4868e0a-f7c6-4d86-b7a6-30bb0540a39f 2024-09-16T08:00:59.664 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:00:59.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC75edmb1shKRAAnyJdmCzIEfx5s55RiDIQ8Q== 2024-09-16T08:00:59.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC75edmb1shKRAAnyJdmCzIEfx5s55RiDIQ8Q=="}' 2024-09-16T08:00:59.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e4868e0a-f7c6-4d86-b7a6-30bb0540a39f -i td/test-erasure-eio/3/new.json 2024-09-16T08:01:01.062 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:01:01.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-16T08:01:01.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC75edmb1shKRAAnyJdmCzIEfx5s55RiDIQ8Q== --osd-uuid e4868e0a-f7c6-4d86-b7a6-30bb0540a39f 2024-09-16T08:01:01.121 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:01:01.117+0000 7f3b133b5880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:01:01.240 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:01:01.237+0000 7f3b133b5880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:01:01.242 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:01:01.241+0000 7f3b133b5880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:01:01.243 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:01:01.241+0000 7f3b133b5880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:01:01.243 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:01:01.241+0000 7f3b133b5880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:01:01.244 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:01:01.241+0000 7f3b133b5880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:01:01.482 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:01:01.481+0000 7f3b133b5880 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-16T08:01:07.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-16T08:01:07.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:01:07.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T08:01:07.529 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-16T08:01:07.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:01:08.096 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-16T08:01:08.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T08:01:08.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:01:08.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:01:08.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:01:08.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:01:08.139 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:01:08.137+0000 7f015a57b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:01:08.159 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:01:08.157+0000 7f015a57b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:01:08.161 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:01:08.157+0000 7f015a57b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:01:08.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T08:01:08.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:01:08.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T08:01:08.638 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:01:08.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:01:08.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:01:08.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:08.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:01:08.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:08.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:01:08.743 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:01:08.741+0000 7f015a57b880 -1 Falling back to public interface 2024-09-16T08:01:09.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:01:10.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:01:10.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:10.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:01:10.124 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:01:10.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:10.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:01:10.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:01:11.485 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:01:11.481+0000 7f015a57b880 -1 osd.3 0 log_to_monitors true 2024-09-16T08:01:11.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:01:11.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:11.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:01:11.541 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:01:11.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:11.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:01:12.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:01:13.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:01:13.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:13.151 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:01:13.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:01:13.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:13.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:01:13.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:01:14.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:01:14.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:14.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:01:14.651 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:01:14.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:14.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:01:15.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:01:16.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:01:16.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:16.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:01:16.179 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:01:16.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:16.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:01:16.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:01:17.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:01:17.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:17.881 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:01:17.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:01:17.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:17.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:01:18.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:01:19.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:01:19.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:19.429 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:01:19.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:01:19.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:19.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:01:19.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:01:20.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:01:20.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:20.919 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:01:20.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:01:20.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:20.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:01:21.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:01:22.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:01:22.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:22.464 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:01:22.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:01:22.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:22.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:01:22.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:01:23.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:01:23.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:23.977 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:01:23.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:01:23.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:23.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:01:24.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:01:25.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:01:25.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:25.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:01:25.554 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:01:25.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:25.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:01:26.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:01:27.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:01:27.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:27.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:01:27.226 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:01:27.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:27.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:01:28.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:01:29.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:01:29.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:29.551 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:01:29.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:01:29.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:29.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:01:30.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:01:31.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:01:31.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:31.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:01:31.561 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:01:31.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:31.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:01:32.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:01:33.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:01:33.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:33.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:01:33.537 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:01:33.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:01:33.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:34.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:01:35.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:01:35.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:35.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:01:35.235 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:01:35.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:35.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:01:36.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:01:37.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:01:37.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:37.527 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:01:37.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:01:37.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:37.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:01:37.993 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2178184936,v1:127.0.0.1:6827/2178184936] [v2:127.0.0.1:6828/2178184936,v1:127.0.0.1:6829/2178184936] exists,up e4868e0a-f7c6-4d86-b7a6-30bb0540a39f 2024-09-16T08:01:37.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:01:37.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:01:37.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:01:37.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:01:37.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-16T08:01:37.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:01:37.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:01:37.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-16T08:01:37.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:01:37.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-16T08:01:37.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-16T08:01:37.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:01:37.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:01:37.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:01:37.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-16T08:01:37.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-16T08:01:37.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:01:37.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:01:37.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:01:37.997 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:01:37.997 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:01:37.997 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:01:37.997 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:01:37.997 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:01:37.997 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:01:37.998 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:01:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:01:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:01:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:01:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:01:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:01:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:01:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:01:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:01:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:01:37.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:01:37.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:01:37.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:01:37.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-16T08:01:37.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:01:38.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3815112f-bb72-4b41-8bfc-122df1d35734 2024-09-16T08:01:38.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 3815112f-bb72-4b41-8bfc-122df1d35734' 2024-09-16T08:01:38.000 INFO:tasks.workunit.client.0.smithi162.stdout:add osd4 3815112f-bb72-4b41-8bfc-122df1d35734 2024-09-16T08:01:38.001 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:01:38.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDi5edmXhP4ARAAH6KL+INWycGYJtfGHR0zYQ== 2024-09-16T08:01:38.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDi5edmXhP4ARAAH6KL+INWycGYJtfGHR0zYQ=="}' 2024-09-16T08:01:38.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3815112f-bb72-4b41-8bfc-122df1d35734 -i td/test-erasure-eio/4/new.json 2024-09-16T08:01:38.634 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:01:38.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-16T08:01:38.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDi5edmXhP4ARAAH6KL+INWycGYJtfGHR0zYQ== --osd-uuid 3815112f-bb72-4b41-8bfc-122df1d35734 2024-09-16T08:01:38.701 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:01:38.696+0000 7ff632150880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:01:38.837 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:01:38.832+0000 7ff632150880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:01:38.839 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:01:38.836+0000 7ff632150880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:01:38.840 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:01:38.836+0000 7ff632150880 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-16T08:01:38.840 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:01:38.836+0000 7ff632150880 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-16T08:01:38.840 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:01:38.836+0000 7ff632150880 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-16T08:01:38.975 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:01:38.972+0000 7ff632150880 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-16T08:01:46.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-16T08:01:46.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:01:46.736 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd4 key to auth repository 2024-09-16T08:01:46.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-16T08:01:46.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:01:47.311 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.4 2024-09-16T08:01:47.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-16T08:01:47.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:01:47.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:01:47.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:01:47.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:01:47.353 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:01:47.348+0000 7fa1ce279880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:01:47.384 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:01:47.380+0000 7fa1ce279880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:01:47.386 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:01:47.384+0000 7fa1ce279880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:01:47.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-16T08:01:47.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:01:47.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-16T08:01:47.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:01:47.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:01:47.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:47.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:01:47.763 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:01:47.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:47.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:01:47.952 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:01:47.948+0000 7fa1ce279880 -1 Falling back to public interface 2024-09-16T08:01:48.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:01:49.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:01:49.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:49.198 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:01:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:01:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:01:49.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:01:50.438 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:01:50.436+0000 7fa1ce279880 -1 osd.4 0 log_to_monitors true 2024-09-16T08:01:50.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:01:50.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:50.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:01:50.684 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:01:50.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:50.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:01:51.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:01:52.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:01:52.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:52.222 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:01:52.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:01:52.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:52.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:01:53.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:01:54.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:01:54.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:54.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:01:54.023 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:01:54.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:54.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:01:54.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:01:55.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:01:55.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:55.618 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:01:55.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:01:55.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:55.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:01:56.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:01:57.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:01:57.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:57.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:01:57.196 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:01:57.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:01:57.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:57.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:01:58.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:01:58.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:01:58.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:01:58.754 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:01:58.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:01:58.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:01:59.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:02:00.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:02:00.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:02:00.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:02:00.334 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:02:00.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:02:00.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:02:00.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:02:01.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:02:01.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:02:01.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:02:01.976 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:02:01.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:02:01.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:02:02.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:02:03.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:02:03.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:02:03.512 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:02:03.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:02:03.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:02:03.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:02:03.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:02:04.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:02:04.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:02:04.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:02:04.965 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:02:04.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:02:04.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:02:05.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:02:06.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:02:06.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:02:06.917 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:02:06.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:02:06.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:02:06.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:02:07.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:02:08.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:02:08.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:02:08.891 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:02:08.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:02:08.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:02:08.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:02:10.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:02:11.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:02:11.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:02:11.044 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:02:11.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:02:11.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:02:11.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:02:12.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:02:13.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:02:13.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:02:13.228 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:02:13.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:02:13.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:02:13.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:02:13.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:02:14.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:02:14.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:02:14.672 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:02:14.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:02:14.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:02:14.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:02:15.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:02:16.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:02:16.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:02:16.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:02:16.364 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:02:16.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:02:16.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:02:16.876 INFO:tasks.workunit.client.0.smithi162.stdout:osd.4 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/305515047,v1:127.0.0.1:6835/305515047] [v2:127.0.0.1:6836/305515047,v1:127.0.0.1:6837/305515047] exists,up 3815112f-bb72-4b41-8bfc-122df1d35734 2024-09-16T08:02:16.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:02:16.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:02:16.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:02:16.877 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-16T08:02:16.877 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T08:02:16.878 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T08:02:16.878 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:02:16.878 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:02:16.878 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:02:16.878 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.0.asok 2024-09-16T08:02:16.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-16T08:02:16.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.0.asok log flush 2024-09-16T08:02:16.971 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T08:02:16.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-16T08:02:16.987 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T07:59:17.383+0000 7f5d0e66e880 0 load: jerasure load: lrc 2024-09-16T08:02:16.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: CEPH_ARGS='--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:02:16.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:621: TEST_ec_recovery_unfound: local poolname=pool-jerasure 2024-09-16T08:02:16.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:622: TEST_ec_recovery_unfound: create_erasure_coded_pool pool-jerasure 3 2 2024-09-16T08:02:16.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T08:02:16.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-16T08:02:16.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-16T08:02:16.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-16T08:02:16.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-16T08:02:16.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-16T08:02:16.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-16T08:02:17.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-16T08:02:17.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-16T08:02:19.436 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-16T08:02:19.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T08:02:20.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-16T08:02:20.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:02:20.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:02:20.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:02:20.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:02:20.448 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:02:20.449 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:02:20.449 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:02:20.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:02:20.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:02:20.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:02:20.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:02:20.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:02:20.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:02:20.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:02:20.669 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:02:21.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:02:21.115 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:02:21.116 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:02:21.116 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T08:02:21.116 INFO:tasks.workunit.client.0.smithi162.stderr:4' 2024-09-16T08:02:21.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:02:21.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:02:21.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:02:21.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803823 2024-09-16T08:02:21.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803823 2024-09-16T08:02:21.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803823' 2024-09-16T08:02:21.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:02:21.432 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:02:21.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607591 2024-09-16T08:02:21.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607591 2024-09-16T08:02:21.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803823 1-51539607591' 2024-09-16T08:02:21.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:02:21.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:02:21.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=77309411362 2024-09-16T08:02:21.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 77309411362 2024-09-16T08:02:21.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803823 1-51539607591 2-77309411362' 2024-09-16T08:02:21.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:02:21.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:02:21.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=120259084315 2024-09-16T08:02:21.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 120259084315 2024-09-16T08:02:21.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803823 1-51539607591 2-77309411362 3-120259084315' 2024-09-16T08:02:21.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:02:21.958 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T08:02:22.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=171798691861 2024-09-16T08:02:22.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 171798691861 2024-09-16T08:02:22.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803823 1-51539607591 2-77309411362 3-120259084315 4-171798691861' 2024-09-16T08:02:22.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:02:22.115 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803823 2024-09-16T08:02:22.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:02:22.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:02:22.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803823 2024-09-16T08:02:22.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:02:22.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803823 2024-09-16T08:02:22.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803823' 2024-09-16T08:02:22.122 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803823 2024-09-16T08:02:22.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:02:22.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803823 -lt 25769803823 2024-09-16T08:02:22.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:02:22.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607591 2024-09-16T08:02:22.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:02:22.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:02:22.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607591 2024-09-16T08:02:22.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:02:22.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607591 2024-09-16T08:02:22.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607591' 2024-09-16T08:02:22.566 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 51539607591 2024-09-16T08:02:22.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:02:23.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607592 -lt 51539607591 2024-09-16T08:02:23.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:02:23.094 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:02:23.094 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-77309411362 2024-09-16T08:02:23.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:02:23.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:02:23.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-77309411362 2024-09-16T08:02:23.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=77309411362 2024-09-16T08:02:23.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 77309411362' 2024-09-16T08:02:23.098 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 77309411362 2024-09-16T08:02:23.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:02:23.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 77309411362 -lt 77309411362 2024-09-16T08:02:23.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:02:23.514 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-120259084315 2024-09-16T08:02:23.514 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:02:23.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:02:23.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-120259084315 2024-09-16T08:02:23.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:02:23.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=120259084315 2024-09-16T08:02:23.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 120259084315' 2024-09-16T08:02:23.520 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 120259084315 2024-09-16T08:02:23.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:02:24.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 120259084313 -lt 120259084315 2024-09-16T08:02:24.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T08:02:25.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T08:02:25.004 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:02:25.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 120259084316 -lt 120259084315 2024-09-16T08:02:25.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:02:25.434 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-171798691861 2024-09-16T08:02:25.435 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:02:25.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T08:02:25.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-171798691861 2024-09-16T08:02:25.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:02:25.439 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 171798691861 2024-09-16T08:02:25.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=171798691861 2024-09-16T08:02:25.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 171798691861' 2024-09-16T08:02:25.440 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T08:02:25.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 171798691861 -lt 171798691861 2024-09-16T08:02:25.860 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:02:25.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:02:25.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:02:26.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:02:26.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:02:26.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:02:26.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:02:26.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:02:26.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:02:26.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:02:26.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:02:26.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:02:26.891 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:02:26.891 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:02:26.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:02:27.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:02:27.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:02:27.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:02:27.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:624: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-09-16T08:02:27.865 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T08:02:27.865 INFO:tasks.workunit.client.0.smithi162.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T08:02:21.100151+0000 0'0 47:13 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-16T08:02:19.028848+0000 0'0 2024-09-16T08:02:19.028848+0000 0 0 periodic scrub scheduled @ 2024-09-17T08:11:37.795139+0000 0 0 2024-09-16T08:02:27.865 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T08:02:20.052193+0000 0'0 47:14 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-16T07:59:08.633562+0000 0'0 2024-09-16T07:59:08.633562+0000 0 0 periodic scrub scheduled @ 2024-09-17T09:27:32.973935+0000 0 0 2024-09-16T08:02:27.865 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T08:02:20.151696+0000 0'0 46:97 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-16T07:59:08.633562+0000 0'0 2024-09-16T07:59:08.633562+0000 0 0 periodic scrub scheduled @ 2024-09-17T13:45:50.347880+0000 0 0 2024-09-16T08:02:27.866 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T08:02:19.967277+0000 0'0 47:14 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-16T07:59:08.633562+0000 0'0 2024-09-16T07:59:08.633562+0000 0 0 periodic scrub scheduled @ 2024-09-17T17:27:05.782477+0000 0 0 2024-09-16T08:02:27.866 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T08:01:00.518182+0000 0'0 47:45 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T07:59:08.633562+0000 0'0 2024-09-16T07:59:08.633562+0000 0 0 periodic scrub scheduled @ 2024-09-17T12:03:57.876807+0000 0 0 2024-09-16T08:02:27.866 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-09-16T08:02:27.866 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T08:02:27.868 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-09-16T08:02:27.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:626: TEST_ec_recovery_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-16T08:02:27.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-16T08:02:27.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-16T08:02:27.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-16T08:02:27.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:02:27.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-16T08:02:27.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:02:27.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-16T08:02:27.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:02:27.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-16T08:02:27.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:02:27.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-16T08:02:27.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:28.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:02:28.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: get_osds pool-jerasure myobject 2024-09-16T08:02:28.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:02:28.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-16T08:02:28.083 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-16T08:02:28.083 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:02:28.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:02:28.539 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:02:28.539 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T08:02:28.539 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T08:02:28.539 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T08:02:28.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2 2024-09-16T08:02:28.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: local -a initial_osds 2024-09-16T08:02:28.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:629: TEST_ec_recovery_unfound: local last_osd=2 2024-09-16T08:02:28.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:630: TEST_ec_recovery_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-16T08:02:28.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:02:28.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:02:28.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:02:28.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:02:28.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:02:28.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:02:28.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:631: TEST_ec_recovery_unfound: ceph osd down 2 2024-09-16T08:02:29.834 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 is already down. 2024-09-16T08:02:29.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:632: TEST_ec_recovery_unfound: ceph osd out 2 2024-09-16T08:02:30.692 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 is already out. 2024-09-16T08:02:30.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:634: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-09-16T08:02:31.129 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T08:02:31.129 INFO:tasks.workunit.client.0.smithi162.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T08:02:21.100151+0000 0'0 47:13 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-16T08:02:19.028848+0000 0'0 2024-09-16T08:02:19.028848+0000 0 0 periodic scrub scheduled @ 2024-09-17T08:11:37.795139+0000 0 0 2024-09-16T08:02:31.129 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T08:02:20.052193+0000 0'0 47:14 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-16T07:59:08.633562+0000 0'0 2024-09-16T07:59:08.633562+0000 0 0 periodic scrub scheduled @ 2024-09-17T09:27:32.973935+0000 0 0 2024-09-16T08:02:31.129 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T08:02:20.151696+0000 0'0 48:99 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-16T07:59:08.633562+0000 0'0 2024-09-16T07:59:08.633562+0000 0 0 periodic scrub scheduled @ 2024-09-17T13:45:50.347880+0000 0 0 2024-09-16T08:02:31.129 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T08:02:19.967277+0000 0'0 47:14 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-16T07:59:08.633562+0000 0'0 2024-09-16T07:59:08.633562+0000 0 0 periodic scrub scheduled @ 2024-09-17T17:27:05.782477+0000 0 0 2024-09-16T08:02:31.129 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T08:01:00.518182+0000 0'0 47:45 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T07:59:08.633562+0000 0'0 2024-09-16T07:59:08.633562+0000 0 0 periodic scrub scheduled @ 2024-09-17T12:03:57.876807+0000 0 0 2024-09-16T08:02:31.130 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-09-16T08:02:31.130 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T08:02:31.132 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-09-16T08:02:31.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:636: TEST_ec_recovery_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2024-09-16T08:02:31.142 INFO:tasks.workunit.client.0.smithi162.stderr:4+0 records in 2024-09-16T08:02:31.142 INFO:tasks.workunit.client.0.smithi162.stderr:4+0 records out 2024-09-16T08:02:31.142 INFO:tasks.workunit.client.0.smithi162.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000296084 s, 13.8 MB/s 2024-09-16T08:02:31.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: seq 1 100 2024-09-16T08:02:31.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:31.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:31.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:31.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:31.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:31.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:32.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:32.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:32.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:32.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:33.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:33.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:33.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:33.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:33.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:33.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:33.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:33.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:34.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:34.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:34.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:34.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:34.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:34.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:35.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:35.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:35.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:35.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:35.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:35.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:35.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:35.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:36.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:36.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:36.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:36.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:36.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:36.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:36.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:36.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:37.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:37.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:37.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:37.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:37.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:37.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:38.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:38.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:38.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:38.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:38.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:38.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:38.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:38.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:39.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:39.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:39.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:39.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:39.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:39.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:39.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:39.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:40.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:40.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:40.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:40.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:40.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:40.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:41.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:41.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:41.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:41.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:41.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:41.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:41.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:41.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:42.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:42.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:42.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:42.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:42.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:42.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:42.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:42.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:43.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:43.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:43.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:43.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:43.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:43.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:44.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:44.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:44.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:44.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:44.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:44.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:45.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:45.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:45.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:45.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:45.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:45.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:45.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:45.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:46.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:46.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:46.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:46.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:46.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:46.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:46.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:46.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:47.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:47.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:47.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:47.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:47.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:47.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:47.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:47.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:48.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:48.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:48.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:48.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:48.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:48.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:49.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:49.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:49.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:49.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:49.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:49.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:49.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:49.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:50.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:50.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:50.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:50.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:50.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:50.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:51.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:51.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:51.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:51.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:51.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:51.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:51.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:51.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:52.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:52.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:52.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:52.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:52.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:52.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:53.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:53.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:53.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:53.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:53.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:53.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:53.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:53.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:54.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:54.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:54.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:54.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:54.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:54.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:54.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:54.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:55.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:55.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:55.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:55.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:55.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:55.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:55.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:55.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:56.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:56.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:56.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:56.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:56.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:56.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:57.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:57.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:57.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:57.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:57.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:57.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:57.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:57.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:58.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:58.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:58.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:58.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:58.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:58.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:58.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:02:58.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2024-09-16T08:02:59.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:642: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 0 2024-09-16T08:02:59.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=ec 2024-09-16T08:02:59.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T08:02:59.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T08:02:59.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T08:02:59.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-jerasure 2024-09-16T08:02:59.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T08:02:59.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj75 2024-09-16T08:02:59.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T08:02:59.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/test-erasure-eio 2024-09-16T08:02:59.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T08:02:59.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T08:02:59.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T08:02:59.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:02:59.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-jerasure obj75 2024-09-16T08:02:59.226 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:02:59.226 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-09-16T08:02:59.226 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj75 2024-09-16T08:02:59.226 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:02:59.707 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:02:59.707 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:02:59.707 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T08:02:59.707 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T08:02:59.707 INFO:tasks.workunit.client.0.smithi162.stderr:2147483647' 2024-09-16T08:02:59.707 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-09-16T08:02:59.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T08:02:59.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T08:02:59.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' ec '!=' ec ']' 2024-09-16T08:02:59.708 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/test-erasure-eio/3/type 2024-09-16T08:02:59.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T08:02:59.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T08:02:59.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T08:02:59.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T08:02:59.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T08:02:59.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T08:02:59.712 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T08:02:59.712 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T08:02:59.712 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T08:02:59.712 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T08:02:59.713 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:02:59.713 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:02:59.713 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:02:59.714 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.3.asok 2024-09-16T08:02:59.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79228/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T08:02:59.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T08:02:59.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T08:02:59.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T08:02:59.806 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T08:02:59.806 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T08:02:59.806 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T08:02:59.806 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:02:59.806 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:02:59.807 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:02:59.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.3.asok 2024-09-16T08:02:59.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T08:02:59.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.3.asok injectdataerr pool-jerasure obj75 0 2024-09-16T08:02:59.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:643: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 1 2024-09-16T08:02:59.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=ec 2024-09-16T08:02:59.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T08:02:59.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T08:02:59.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T08:02:59.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-jerasure 2024-09-16T08:02:59.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T08:02:59.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj75 2024-09-16T08:02:59.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T08:02:59.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/test-erasure-eio 2024-09-16T08:02:59.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T08:02:59.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=1 2024-09-16T08:02:59.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T08:02:59.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:02:59.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-jerasure obj75 2024-09-16T08:02:59.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:02:59.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-09-16T08:02:59.914 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj75 2024-09-16T08:02:59.914 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:03:00.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:03:00.328 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:03:00.329 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T08:03:00.329 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T08:03:00.329 INFO:tasks.workunit.client.0.smithi162.stderr:2147483647' 2024-09-16T08:03:00.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-09-16T08:03:00.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T08:03:00.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=1 2024-09-16T08:03:00.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' ec '!=' ec ']' 2024-09-16T08:03:00.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/test-erasure-eio/1/type 2024-09-16T08:03:00.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T08:03:00.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-16T08:03:00.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T08:03:00.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-16T08:03:00.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T08:03:00.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T08:03:00.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T08:03:00.334 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-16T08:03:00.334 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T08:03:00.334 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T08:03:00.334 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:03:00.334 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:03:00.334 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:03:00.335 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.1.asok 2024-09-16T08:03:00.335 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79228/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-16T08:03:00.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T08:03:00.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T08:03:00.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T08:03:00.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.1 2024-09-16T08:03:00.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T08:03:00.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T08:03:00.430 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:03:00.430 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:03:00.431 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:03:00.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.1.asok 2024-09-16T08:03:00.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T08:03:00.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.1.asok injectdataerr pool-jerasure obj75 1 2024-09-16T08:03:00.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:645: TEST_ec_recovery_unfound: activate_osd td/test-erasure-eio 2 2024-09-16T08:03:00.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-16T08:03:00.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T08:03:00.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T08:03:00.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T08:03:00.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-16T08:03:00.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:03:00.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:03:00.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:03:00.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:03:00.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-16T08:03:00.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-16T08:03:00.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T08:03:00.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T08:03:00.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:03:00.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T08:03:00.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:03:00.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:03:00.537 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:03:00.537 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:03:00.537 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:03:00.537 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:03:00.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:03:00.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:03:00.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:03:00.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:03:00.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:03:00.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:03:00.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:03:00.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:03:00.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T08:03:00.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T08:03:00.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-16T08:03:00.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T08:03:00.538 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-16T08:03:00.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:03:00.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-16T08:03:00.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T08:03:00.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T08:03:00.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T08:03:00.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T08:03:00.570 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:03:00.566+0000 7fcbbfb87880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:03:00.573 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:03:00.570+0000 7fcbbfb87880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:03:00.580 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:03:00.570+0000 7fcbbfb87880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:03:00.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T08:03:00.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:03:00.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T08:03:00.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:03:00.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:03:00.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:03:00.978 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:03:00.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:03:00.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:03:00.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:03:01.116 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:03:01.110+0000 7fcbbfb87880 -1 Falling back to public interface 2024-09-16T08:03:01.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:03:02.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:03:02.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:03:02.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:03:02.384 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:03:02.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:03:02.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:03:02.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:03:03.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:03:03.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:03:03.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:03:03.848 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:03:03.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:03:03.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:03:04.213 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:03:04.206+0000 7fcbbfb87880 -1 osd.2 47 log_to_monitors true 2024-09-16T08:03:04.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:03:05.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:03:05.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:03:05.334 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:03:05.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:03:05.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:03:05.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:03:05.978 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up out weight 0 up_from 55 up_thru 19 down_at 48 last_clean_interval [18,47) [v2:127.0.0.1:6818/312801001,v1:127.0.0.1:6819/312801001] [v2:127.0.0.1:6820/312801001,v1:127.0.0.1:6821/312801001] exists,up 90367fc6-7334-4796-9ee0-844712945b4c 2024-09-16T08:03:05.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:03:05.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:03:05.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:03:05.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:646: TEST_ec_recovery_unfound: ceph osd in 2 2024-09-16T08:03:06.900 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 is already in. 2024-09-16T08:03:06.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:648: TEST_ec_recovery_unfound: sleep 15 2024-09-16T08:03:21.919 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:650: TEST_ec_recovery_unfound: seq 1 100 2024-09-16T08:03:21.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:650: TEST_ec_recovery_unfound: for tmp in $(seq 1 100) 2024-09-16T08:03:21.921 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: get_state 2.0 2024-09-16T08:03:21.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-16T08:03:21.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-16T08:03:21.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-16T08:03:21.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-16T08:03:22.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: state=active+recovery_unfound+degraded 2024-09-16T08:03:22.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:652: TEST_ec_recovery_unfound: echo active+recovery_unfound+degraded 2024-09-16T08:03:22.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:652: TEST_ec_recovery_unfound: grep recovery_unfound 2024-09-16T08:03:22.338 INFO:tasks.workunit.client.0.smithi162.stdout:active+recovery_unfound+degraded 2024-09-16T08:03:22.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' 0 = 0 ']' 2024-09-16T08:03:22.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:654: TEST_ec_recovery_unfound: break 2024-09-16T08:03:22.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:660: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-09-16T08:03:22.707 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-16T08:03:22.707 INFO:tasks.workunit.client.0.smithi162.stdout:2.0 101 1 3 0 1 413696 0 0 101 0 101 active+recovery_unfound+degraded 2024-09-16T08:03:11.016711+0000 53'101 60:349 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-16T08:02:19.028848+0000 0'0 2024-09-16T08:02:19.028848+0000 0 0 periodic scrub scheduled @ 2024-09-17T18:39:40.559784+0000 0 0 2024-09-16T08:03:22.707 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T08:03:08.361516+0000 0'0 60:50 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-16T07:59:08.633562+0000 0'0 2024-09-16T07:59:08.633562+0000 0 0 periodic scrub scheduled @ 2024-09-17T19:00:08.374187+0000 0 0 2024-09-16T08:03:22.707 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T08:02:20.151696+0000 0'0 59:110 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-16T07:59:08.633562+0000 0'0 2024-09-16T07:59:08.633562+0000 0 0 periodic scrub scheduled @ 2024-09-17T13:45:50.347880+0000 0 0 2024-09-16T08:03:22.708 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T08:02:19.967277+0000 0'0 59:26 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-16T07:59:08.633562+0000 0'0 2024-09-16T07:59:08.633562+0000 0 0 periodic scrub scheduled @ 2024-09-17T17:27:05.782477+0000 0 0 2024-09-16T08:03:22.708 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-16T08:03:07.881445+0000 0'0 60:82 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-16T07:59:08.633562+0000 0'0 2024-09-16T07:59:08.633562+0000 0 0 periodic scrub scheduled @ 2024-09-17T10:10:43.772778+0000 0 0 2024-09-16T08:03:22.708 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-09-16T08:03:22.708 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-16T08:03:22.710 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-09-16T08:03:22.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:661: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-16T08:03:22.862 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-09-16T08:03:22.862 INFO:tasks.workunit.client.0.smithi162.stdout: "num_missing": 1, 2024-09-16T08:03:22.863 INFO:tasks.workunit.client.0.smithi162.stdout: "num_unfound": 1, 2024-09-16T08:03:22.863 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": [ 2024-09-16T08:03:22.863 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T08:03:22.863 INFO:tasks.workunit.client.0.smithi162.stdout: "oid": { 2024-09-16T08:03:22.863 INFO:tasks.workunit.client.0.smithi162.stdout: "oid": "obj75", 2024-09-16T08:03:22.863 INFO:tasks.workunit.client.0.smithi162.stdout: "key": "", 2024-09-16T08:03:22.863 INFO:tasks.workunit.client.0.smithi162.stdout: "snapid": -2, 2024-09-16T08:03:22.863 INFO:tasks.workunit.client.0.smithi162.stdout: "hash": 3565005477, 2024-09-16T08:03:22.863 INFO:tasks.workunit.client.0.smithi162.stdout: "max": 0, 2024-09-16T08:03:22.863 INFO:tasks.workunit.client.0.smithi162.stdout: "pool": 2, 2024-09-16T08:03:22.863 INFO:tasks.workunit.client.0.smithi162.stdout: "namespace": "" 2024-09-16T08:03:22.863 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:22.863 INFO:tasks.workunit.client.0.smithi162.stdout: "need": "53'76", 2024-09-16T08:03:22.864 INFO:tasks.workunit.client.0.smithi162.stdout: "have": "0'0", 2024-09-16T08:03:22.864 INFO:tasks.workunit.client.0.smithi162.stdout: "flags": "none", 2024-09-16T08:03:22.864 INFO:tasks.workunit.client.0.smithi162.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-16T08:03:22.864 INFO:tasks.workunit.client.0.smithi162.stdout: "locations": [ 2024-09-16T08:03:22.864 INFO:tasks.workunit.client.0.smithi162.stdout: "0(3)", 2024-09-16T08:03:22.864 INFO:tasks.workunit.client.0.smithi162.stdout: "4(2)" 2024-09-16T08:03:22.864 INFO:tasks.workunit.client.0.smithi162.stdout: ] 2024-09-16T08:03:22.864 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T08:03:22.864 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:22.864 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "NotRecovering", 2024-09-16T08:03:22.864 INFO:tasks.workunit.client.0.smithi162.stdout: "available_might_have_unfound": true, 2024-09-16T08:03:22.864 INFO:tasks.workunit.client.0.smithi162.stdout: "might_have_unfound": [], 2024-09-16T08:03:22.864 INFO:tasks.workunit.client.0.smithi162.stdout: "more": false 2024-09-16T08:03:22.865 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-09-16T08:03:22.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:662: TEST_ec_recovery_unfound: ceph pg 2.0 query 2024-09-16T08:03:23.015 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-09-16T08:03:23.015 INFO:tasks.workunit.client.0.smithi162.stdout: "snap_trimq": "[]", 2024-09-16T08:03:23.015 INFO:tasks.workunit.client.0.smithi162.stdout: "snap_trimq_len": 0, 2024-09-16T08:03:23.015 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "active+recovery_unfound+degraded", 2024-09-16T08:03:23.015 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch": 60, 2024-09-16T08:03:23.015 INFO:tasks.workunit.client.0.smithi162.stdout: "up": [ 2024-09-16T08:03:23.015 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-16T08:03:23.015 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-16T08:03:23.015 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-16T08:03:23.015 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-16T08:03:23.016 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-09-16T08:03:23.016 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:23.016 INFO:tasks.workunit.client.0.smithi162.stdout: "acting": [ 2024-09-16T08:03:23.016 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-16T08:03:23.016 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-16T08:03:23.016 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-16T08:03:23.016 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-16T08:03:23.016 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-09-16T08:03:23.016 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:23.016 INFO:tasks.workunit.client.0.smithi162.stdout: "acting_recovery_backfill": [ 2024-09-16T08:03:23.016 INFO:tasks.workunit.client.0.smithi162.stdout: "0(3)", 2024-09-16T08:03:23.017 INFO:tasks.workunit.client.0.smithi162.stdout: "1(1)", 2024-09-16T08:03:23.017 INFO:tasks.workunit.client.0.smithi162.stdout: "2(4)", 2024-09-16T08:03:23.017 INFO:tasks.workunit.client.0.smithi162.stdout: "3(0)", 2024-09-16T08:03:23.017 INFO:tasks.workunit.client.0.smithi162.stdout: "4(2)" 2024-09-16T08:03:23.017 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:23.017 INFO:tasks.workunit.client.0.smithi162.stdout: "info": { 2024-09-16T08:03:23.017 INFO:tasks.workunit.client.0.smithi162.stdout: "pgid": "2.0s0", 2024-09-16T08:03:23.017 INFO:tasks.workunit.client.0.smithi162.stdout: "last_update": "53'101", 2024-09-16T08:03:23.017 INFO:tasks.workunit.client.0.smithi162.stdout: "last_complete": "53'75", 2024-09-16T08:03:23.017 INFO:tasks.workunit.client.0.smithi162.stdout: "log_tail": "0'0", 2024-09-16T08:03:23.017 INFO:tasks.workunit.client.0.smithi162.stdout: "last_user_version": 101, 2024-09-16T08:03:23.017 INFO:tasks.workunit.client.0.smithi162.stdout: "last_backfill": "MAX", 2024-09-16T08:03:23.018 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [], 2024-09-16T08:03:23.018 INFO:tasks.workunit.client.0.smithi162.stdout: "history": { 2024-09-16T08:03:23.018 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_created": 45, 2024-09-16T08:03:23.018 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_pool_created": 45, 2024-09-16T08:03:23.018 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 60, 2024-09-16T08:03:23.018 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_started": 58, 2024-09-16T08:03:23.018 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 46, 2024-09-16T08:03:23.018 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_clean": 45, 2024-09-16T08:03:23.018 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_split": 0, 2024-09-16T08:03:23.018 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_marked_full": 0, 2024-09-16T08:03:23.018 INFO:tasks.workunit.client.0.smithi162.stdout: "same_up_since": 58, 2024-09-16T08:03:23.018 INFO:tasks.workunit.client.0.smithi162.stdout: "same_interval_since": 58, 2024-09-16T08:03:23.019 INFO:tasks.workunit.client.0.smithi162.stdout: "same_primary_since": 45, 2024-09-16T08:03:23.019 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-16T08:03:23.019 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.019 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-16T08:03:23.019 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.019 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.019 INFO:tasks.workunit.client.0.smithi162.stdout: "prior_readable_until_ub": 0 2024-09-16T08:03:23.019 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.019 INFO:tasks.workunit.client.0.smithi162.stdout: "stats": { 2024-09-16T08:03:23.019 INFO:tasks.workunit.client.0.smithi162.stdout: "version": "53'101", 2024-09-16T08:03:23.019 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_seq": 350, 2024-09-16T08:03:23.019 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_epoch": 60, 2024-09-16T08:03:23.019 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "active+recovery_unfound+degraded", 2024-09-16T08:03:23.019 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fresh": "2024-09-16T08:03:11.016711+0000", 2024-09-16T08:03:23.020 INFO:tasks.workunit.client.0.smithi162.stdout: "last_change": "2024-09-16T08:03:11.016711+0000", 2024-09-16T08:03:23.020 INFO:tasks.workunit.client.0.smithi162.stdout: "last_active": "2024-09-16T08:03:11.016711+0000", 2024-09-16T08:03:23.020 INFO:tasks.workunit.client.0.smithi162.stdout: "last_peered": "2024-09-16T08:03:11.016711+0000", 2024-09-16T08:03:23.020 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean": "2024-09-16T08:02:28.075313+0000", 2024-09-16T08:03:23.020 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_active": "2024-09-16T08:03:08.271823+0000", 2024-09-16T08:03:23.020 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_peered": "2024-09-16T08:03:08.271823+0000", 2024-09-16T08:03:23.020 INFO:tasks.workunit.client.0.smithi162.stdout: "last_unstale": "2024-09-16T08:03:11.016711+0000", 2024-09-16T08:03:23.020 INFO:tasks.workunit.client.0.smithi162.stdout: "last_undegraded": "2024-09-16T08:03:07.917441+0000", 2024-09-16T08:03:23.020 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fullsized": "2024-09-16T08:03:11.016711+0000", 2024-09-16T08:03:23.020 INFO:tasks.workunit.client.0.smithi162.stdout: "mapping_epoch": 58, 2024-09-16T08:03:23.020 INFO:tasks.workunit.client.0.smithi162.stdout: "log_start": "0'0", 2024-09-16T08:03:23.021 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_start": "0'0", 2024-09-16T08:03:23.021 INFO:tasks.workunit.client.0.smithi162.stdout: "created": 45, 2024-09-16T08:03:23.021 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 46, 2024-09-16T08:03:23.021 INFO:tasks.workunit.client.0.smithi162.stdout: "parent": "0.0", 2024-09-16T08:03:23.021 INFO:tasks.workunit.client.0.smithi162.stdout: "parent_split_bits": 0, 2024-09-16T08:03:23.021 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-16T08:03:23.021 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.021 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-16T08:03:23.021 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.021 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.021 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_scrubbed": 0, 2024-09-16T08:03:23.021 INFO:tasks.workunit.client.0.smithi162.stdout: "log_size": 101, 2024-09-16T08:03:23.021 INFO:tasks.workunit.client.0.smithi162.stdout: "log_dups_size": 0, 2024-09-16T08:03:23.022 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_size": 101, 2024-09-16T08:03:23.022 INFO:tasks.workunit.client.0.smithi162.stdout: "stats_invalid": false, 2024-09-16T08:03:23.022 INFO:tasks.workunit.client.0.smithi162.stdout: "dirty_stats_invalid": false, 2024-09-16T08:03:23.022 INFO:tasks.workunit.client.0.smithi162.stdout: "omap_stats_invalid": false, 2024-09-16T08:03:23.022 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_stats_invalid": false, 2024-09-16T08:03:23.022 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_bytes_stats_invalid": false, 2024-09-16T08:03:23.022 INFO:tasks.workunit.client.0.smithi162.stdout: "pin_stats_invalid": false, 2024-09-16T08:03:23.022 INFO:tasks.workunit.client.0.smithi162.stdout: "manifest_stats_invalid": false, 2024-09-16T08:03:23.022 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrimq_len": 0, 2024-09-16T08:03:23.022 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_duration": 0, 2024-09-16T08:03:23.022 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-17T18:39:40.559784+0000", 2024-09-16T08:03:23.022 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_duration": 0, 2024-09-16T08:03:23.023 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_trimmed": 0, 2024-09-16T08:03:23.023 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrim_duration": 0, 2024-09-16T08:03:23.023 INFO:tasks.workunit.client.0.smithi162.stdout: "stat_sum": { 2024-09-16T08:03:23.023 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes": 413696, 2024-09-16T08:03:23.023 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects": 101, 2024-09-16T08:03:23.023 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_clones": 0, 2024-09-16T08:03:23.023 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_copies": 505, 2024-09-16T08:03:23.023 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing_on_primary": 1, 2024-09-16T08:03:23.023 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing": 0, 2024-09-16T08:03:23.023 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_degraded": 3, 2024-09-16T08:03:23.023 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_misplaced": 0, 2024-09-16T08:03:23.023 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_unfound": 1, 2024-09-16T08:03:23.023 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_dirty": 101, 2024-09-16T08:03:23.024 INFO:tasks.workunit.client.0.smithi162.stdout: "num_whiteouts": 0, 2024-09-16T08:03:23.024 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read": 0, 2024-09-16T08:03:23.024 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read_kb": 0, 2024-09-16T08:03:23.024 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write": 101, 2024-09-16T08:03:23.024 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write_kb": 404, 2024-09-16T08:03:23.024 INFO:tasks.workunit.client.0.smithi162.stdout: "num_scrub_errors": 0, 2024-09-16T08:03:23.024 INFO:tasks.workunit.client.0.smithi162.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T08:03:23.024 INFO:tasks.workunit.client.0.smithi162.stdout: "num_deep_scrub_errors": 0, 2024-09-16T08:03:23.024 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_recovered": 96, 2024-09-16T08:03:23.024 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_recovered": 393216, 2024-09-16T08:03:23.024 INFO:tasks.workunit.client.0.smithi162.stdout: "num_keys_recovered": 0, 2024-09-16T08:03:23.024 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_omap": 0, 2024-09-16T08:03:23.025 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T08:03:23.025 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T08:03:23.025 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush": 0, 2024-09-16T08:03:23.025 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_kb": 0, 2024-09-16T08:03:23.025 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict": 0, 2024-09-16T08:03:23.025 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_kb": 0, 2024-09-16T08:03:23.025 INFO:tasks.workunit.client.0.smithi162.stdout: "num_promote": 0, 2024-09-16T08:03:23.025 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_high": 0, 2024-09-16T08:03:23.025 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_low": 0, 2024-09-16T08:03:23.025 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_some": 0, 2024-09-16T08:03:23.025 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_full": 0, 2024-09-16T08:03:23.025 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_pinned": 0, 2024-09-16T08:03:23.025 INFO:tasks.workunit.client.0.smithi162.stdout: "num_legacy_snapsets": 0, 2024-09-16T08:03:23.026 INFO:tasks.workunit.client.0.smithi162.stdout: "num_large_omap_objects": 0, 2024-09-16T08:03:23.026 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_manifest": 0, 2024-09-16T08:03:23.026 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_bytes": 0, 2024-09-16T08:03:23.026 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_keys": 0, 2024-09-16T08:03:23.026 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_repaired": 0 2024-09-16T08:03:23.026 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.026 INFO:tasks.workunit.client.0.smithi162.stdout: "up": [ 2024-09-16T08:03:23.026 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-16T08:03:23.026 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-16T08:03:23.026 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-16T08:03:23.026 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-16T08:03:23.026 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-09-16T08:03:23.026 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:23.027 INFO:tasks.workunit.client.0.smithi162.stdout: "acting": [ 2024-09-16T08:03:23.027 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-16T08:03:23.027 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-16T08:03:23.027 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-16T08:03:23.027 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-16T08:03:23.027 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-09-16T08:03:23.027 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:23.027 INFO:tasks.workunit.client.0.smithi162.stdout: "avail_no_missing": [ 2024-09-16T08:03:23.027 INFO:tasks.workunit.client.0.smithi162.stdout: "0(3)", 2024-09-16T08:03:23.027 INFO:tasks.workunit.client.0.smithi162.stdout: "4(2)" 2024-09-16T08:03:23.027 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:23.027 INFO:tasks.workunit.client.0.smithi162.stdout: "object_location_counts": [ 2024-09-16T08:03:23.027 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T08:03:23.027 INFO:tasks.workunit.client.0.smithi162.stdout: "shards": "0(3),1(1),2(4),3(0),4(2)", 2024-09-16T08:03:23.028 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": 100 2024-09-16T08:03:23.028 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.028 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T08:03:23.028 INFO:tasks.workunit.client.0.smithi162.stdout: "shards": "0(3),4(2)", 2024-09-16T08:03:23.028 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": 1 2024-09-16T08:03:23.028 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T08:03:23.028 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:23.028 INFO:tasks.workunit.client.0.smithi162.stdout: "blocked_by": [], 2024-09-16T08:03:23.028 INFO:tasks.workunit.client.0.smithi162.stdout: "up_primary": 3, 2024-09-16T08:03:23.028 INFO:tasks.workunit.client.0.smithi162.stdout: "acting_primary": 3, 2024-09-16T08:03:23.028 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [] 2024-09-16T08:03:23.028 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.029 INFO:tasks.workunit.client.0.smithi162.stdout: "empty": 0, 2024-09-16T08:03:23.029 INFO:tasks.workunit.client.0.smithi162.stdout: "dne": 0, 2024-09-16T08:03:23.029 INFO:tasks.workunit.client.0.smithi162.stdout: "incomplete": 0, 2024-09-16T08:03:23.029 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 60, 2024-09-16T08:03:23.029 INFO:tasks.workunit.client.0.smithi162.stdout: "hit_set_history": { 2024-09-16T08:03:23.029 INFO:tasks.workunit.client.0.smithi162.stdout: "current_last_update": "0'0", 2024-09-16T08:03:23.029 INFO:tasks.workunit.client.0.smithi162.stdout: "history": [] 2024-09-16T08:03:23.029 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T08:03:23.029 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.029 INFO:tasks.workunit.client.0.smithi162.stdout: "peer_info": [ 2024-09-16T08:03:23.029 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T08:03:23.029 INFO:tasks.workunit.client.0.smithi162.stdout: "peer": "0(3)", 2024-09-16T08:03:23.029 INFO:tasks.workunit.client.0.smithi162.stdout: "pgid": "2.0s3", 2024-09-16T08:03:23.030 INFO:tasks.workunit.client.0.smithi162.stdout: "last_update": "53'101", 2024-09-16T08:03:23.030 INFO:tasks.workunit.client.0.smithi162.stdout: "last_complete": "53'101", 2024-09-16T08:03:23.030 INFO:tasks.workunit.client.0.smithi162.stdout: "log_tail": "0'0", 2024-09-16T08:03:23.030 INFO:tasks.workunit.client.0.smithi162.stdout: "last_user_version": 101, 2024-09-16T08:03:23.030 INFO:tasks.workunit.client.0.smithi162.stdout: "last_backfill": "MAX", 2024-09-16T08:03:23.030 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [], 2024-09-16T08:03:23.030 INFO:tasks.workunit.client.0.smithi162.stdout: "history": { 2024-09-16T08:03:23.030 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_created": 45, 2024-09-16T08:03:23.030 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_pool_created": 45, 2024-09-16T08:03:23.030 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 60, 2024-09-16T08:03:23.030 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_started": 58, 2024-09-16T08:03:23.030 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 46, 2024-09-16T08:03:23.030 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_clean": 45, 2024-09-16T08:03:23.031 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_split": 0, 2024-09-16T08:03:23.031 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_marked_full": 0, 2024-09-16T08:03:23.031 INFO:tasks.workunit.client.0.smithi162.stdout: "same_up_since": 58, 2024-09-16T08:03:23.031 INFO:tasks.workunit.client.0.smithi162.stdout: "same_interval_since": 58, 2024-09-16T08:03:23.031 INFO:tasks.workunit.client.0.smithi162.stdout: "same_primary_since": 45, 2024-09-16T08:03:23.031 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-16T08:03:23.031 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.031 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-16T08:03:23.031 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.031 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.031 INFO:tasks.workunit.client.0.smithi162.stdout: "prior_readable_until_ub": 0 2024-09-16T08:03:23.031 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.031 INFO:tasks.workunit.client.0.smithi162.stdout: "stats": { 2024-09-16T08:03:23.031 INFO:tasks.workunit.client.0.smithi162.stdout: "version": "53'101", 2024-09-16T08:03:23.032 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_seq": 129, 2024-09-16T08:03:23.032 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_epoch": 53, 2024-09-16T08:03:23.032 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "active+undersized+degraded", 2024-09-16T08:03:23.032 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fresh": "2024-09-16T08:02:59.217288+0000", 2024-09-16T08:03:23.032 INFO:tasks.workunit.client.0.smithi162.stdout: "last_change": "2024-09-16T08:02:29.869256+0000", 2024-09-16T08:03:23.032 INFO:tasks.workunit.client.0.smithi162.stdout: "last_active": "2024-09-16T08:02:59.217288+0000", 2024-09-16T08:03:23.032 INFO:tasks.workunit.client.0.smithi162.stdout: "last_peered": "2024-09-16T08:02:59.217288+0000", 2024-09-16T08:03:23.032 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean": "2024-09-16T08:02:28.075313+0000", 2024-09-16T08:03:23.032 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_active": "2024-09-16T08:02:29.869256+0000", 2024-09-16T08:03:23.032 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_peered": "2024-09-16T08:02:29.869256+0000", 2024-09-16T08:03:23.032 INFO:tasks.workunit.client.0.smithi162.stdout: "last_unstale": "2024-09-16T08:02:59.217288+0000", 2024-09-16T08:03:23.032 INFO:tasks.workunit.client.0.smithi162.stdout: "last_undegraded": "2024-09-16T08:02:29.525093+0000", 2024-09-16T08:03:23.032 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fullsized": "2024-09-16T08:02:29.524641+0000", 2024-09-16T08:03:23.033 INFO:tasks.workunit.client.0.smithi162.stdout: "mapping_epoch": 58, 2024-09-16T08:03:23.033 INFO:tasks.workunit.client.0.smithi162.stdout: "log_start": "0'0", 2024-09-16T08:03:23.033 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_start": "0'0", 2024-09-16T08:03:23.033 INFO:tasks.workunit.client.0.smithi162.stdout: "created": 45, 2024-09-16T08:03:23.033 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 46, 2024-09-16T08:03:23.033 INFO:tasks.workunit.client.0.smithi162.stdout: "parent": "0.0", 2024-09-16T08:03:23.033 INFO:tasks.workunit.client.0.smithi162.stdout: "parent_split_bits": 0, 2024-09-16T08:03:23.033 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-16T08:03:23.033 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.033 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-16T08:03:23.033 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.033 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.033 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_scrubbed": 0, 2024-09-16T08:03:23.034 INFO:tasks.workunit.client.0.smithi162.stdout: "log_size": 101, 2024-09-16T08:03:23.034 INFO:tasks.workunit.client.0.smithi162.stdout: "log_dups_size": 0, 2024-09-16T08:03:23.034 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_size": 101, 2024-09-16T08:03:23.034 INFO:tasks.workunit.client.0.smithi162.stdout: "stats_invalid": false, 2024-09-16T08:03:23.034 INFO:tasks.workunit.client.0.smithi162.stdout: "dirty_stats_invalid": false, 2024-09-16T08:03:23.034 INFO:tasks.workunit.client.0.smithi162.stdout: "omap_stats_invalid": false, 2024-09-16T08:03:23.034 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_stats_invalid": false, 2024-09-16T08:03:23.034 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_bytes_stats_invalid": false, 2024-09-16T08:03:23.034 INFO:tasks.workunit.client.0.smithi162.stdout: "pin_stats_invalid": false, 2024-09-16T08:03:23.034 INFO:tasks.workunit.client.0.smithi162.stdout: "manifest_stats_invalid": false, 2024-09-16T08:03:23.034 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrimq_len": 0, 2024-09-16T08:03:23.034 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_duration": 0, 2024-09-16T08:03:23.034 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-17T11:41:10.503212+0000", 2024-09-16T08:03:23.035 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_duration": 0, 2024-09-16T08:03:23.035 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_trimmed": 0, 2024-09-16T08:03:23.035 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrim_duration": 0, 2024-09-16T08:03:23.035 INFO:tasks.workunit.client.0.smithi162.stdout: "stat_sum": { 2024-09-16T08:03:23.035 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes": 413696, 2024-09-16T08:03:23.035 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects": 101, 2024-09-16T08:03:23.035 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_clones": 0, 2024-09-16T08:03:23.035 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_copies": 505, 2024-09-16T08:03:23.035 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T08:03:23.035 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing": 0, 2024-09-16T08:03:23.035 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_degraded": 101, 2024-09-16T08:03:23.035 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_misplaced": 0, 2024-09-16T08:03:23.035 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_unfound": 0, 2024-09-16T08:03:23.036 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_dirty": 101, 2024-09-16T08:03:23.036 INFO:tasks.workunit.client.0.smithi162.stdout: "num_whiteouts": 0, 2024-09-16T08:03:23.036 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read": 0, 2024-09-16T08:03:23.036 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read_kb": 0, 2024-09-16T08:03:23.036 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write": 101, 2024-09-16T08:03:23.036 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write_kb": 404, 2024-09-16T08:03:23.036 INFO:tasks.workunit.client.0.smithi162.stdout: "num_scrub_errors": 0, 2024-09-16T08:03:23.036 INFO:tasks.workunit.client.0.smithi162.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T08:03:23.036 INFO:tasks.workunit.client.0.smithi162.stdout: "num_deep_scrub_errors": 0, 2024-09-16T08:03:23.036 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_recovered": 0, 2024-09-16T08:03:23.036 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_recovered": 0, 2024-09-16T08:03:23.036 INFO:tasks.workunit.client.0.smithi162.stdout: "num_keys_recovered": 0, 2024-09-16T08:03:23.036 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_omap": 0, 2024-09-16T08:03:23.037 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T08:03:23.037 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T08:03:23.037 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush": 0, 2024-09-16T08:03:23.037 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_kb": 0, 2024-09-16T08:03:23.037 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict": 0, 2024-09-16T08:03:23.037 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_kb": 0, 2024-09-16T08:03:23.037 INFO:tasks.workunit.client.0.smithi162.stdout: "num_promote": 0, 2024-09-16T08:03:23.037 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_high": 0, 2024-09-16T08:03:23.037 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_low": 0, 2024-09-16T08:03:23.037 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_some": 0, 2024-09-16T08:03:23.037 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_full": 0, 2024-09-16T08:03:23.037 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_pinned": 0, 2024-09-16T08:03:23.037 INFO:tasks.workunit.client.0.smithi162.stdout: "num_legacy_snapsets": 0, 2024-09-16T08:03:23.037 INFO:tasks.workunit.client.0.smithi162.stdout: "num_large_omap_objects": 0, 2024-09-16T08:03:23.038 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_manifest": 0, 2024-09-16T08:03:23.038 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_bytes": 0, 2024-09-16T08:03:23.038 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_keys": 0, 2024-09-16T08:03:23.038 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_repaired": 0 2024-09-16T08:03:23.038 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.038 INFO:tasks.workunit.client.0.smithi162.stdout: "up": [ 2024-09-16T08:03:23.038 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-16T08:03:23.038 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-16T08:03:23.038 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-16T08:03:23.038 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-16T08:03:23.038 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-09-16T08:03:23.038 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:23.038 INFO:tasks.workunit.client.0.smithi162.stdout: "acting": [ 2024-09-16T08:03:23.038 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-16T08:03:23.039 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-16T08:03:23.039 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-16T08:03:23.039 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-16T08:03:23.039 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-09-16T08:03:23.039 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:23.039 INFO:tasks.workunit.client.0.smithi162.stdout: "avail_no_missing": [ 2024-09-16T08:03:23.039 INFO:tasks.workunit.client.0.smithi162.stdout: "3(0)", 2024-09-16T08:03:23.039 INFO:tasks.workunit.client.0.smithi162.stdout: "0(3)", 2024-09-16T08:03:23.039 INFO:tasks.workunit.client.0.smithi162.stdout: "1(1)", 2024-09-16T08:03:23.039 INFO:tasks.workunit.client.0.smithi162.stdout: "4(2)" 2024-09-16T08:03:23.039 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:23.039 INFO:tasks.workunit.client.0.smithi162.stdout: "object_location_counts": [ 2024-09-16T08:03:23.039 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T08:03:23.040 INFO:tasks.workunit.client.0.smithi162.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-16T08:03:23.040 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": 101 2024-09-16T08:03:23.040 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T08:03:23.040 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:23.040 INFO:tasks.workunit.client.0.smithi162.stdout: "blocked_by": [], 2024-09-16T08:03:23.040 INFO:tasks.workunit.client.0.smithi162.stdout: "up_primary": 3, 2024-09-16T08:03:23.040 INFO:tasks.workunit.client.0.smithi162.stdout: "acting_primary": 3, 2024-09-16T08:03:23.040 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [] 2024-09-16T08:03:23.040 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.040 INFO:tasks.workunit.client.0.smithi162.stdout: "empty": 0, 2024-09-16T08:03:23.040 INFO:tasks.workunit.client.0.smithi162.stdout: "dne": 0, 2024-09-16T08:03:23.040 INFO:tasks.workunit.client.0.smithi162.stdout: "incomplete": 0, 2024-09-16T08:03:23.040 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 60, 2024-09-16T08:03:23.040 INFO:tasks.workunit.client.0.smithi162.stdout: "hit_set_history": { 2024-09-16T08:03:23.041 INFO:tasks.workunit.client.0.smithi162.stdout: "current_last_update": "0'0", 2024-09-16T08:03:23.041 INFO:tasks.workunit.client.0.smithi162.stdout: "history": [] 2024-09-16T08:03:23.041 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T08:03:23.041 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.041 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T08:03:23.041 INFO:tasks.workunit.client.0.smithi162.stdout: "peer": "1(1)", 2024-09-16T08:03:23.041 INFO:tasks.workunit.client.0.smithi162.stdout: "pgid": "2.0s1", 2024-09-16T08:03:23.041 INFO:tasks.workunit.client.0.smithi162.stdout: "last_update": "53'101", 2024-09-16T08:03:23.041 INFO:tasks.workunit.client.0.smithi162.stdout: "last_complete": "53'101", 2024-09-16T08:03:23.041 INFO:tasks.workunit.client.0.smithi162.stdout: "log_tail": "0'0", 2024-09-16T08:03:23.041 INFO:tasks.workunit.client.0.smithi162.stdout: "last_user_version": 101, 2024-09-16T08:03:23.041 INFO:tasks.workunit.client.0.smithi162.stdout: "last_backfill": "MAX", 2024-09-16T08:03:23.041 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [], 2024-09-16T08:03:23.042 INFO:tasks.workunit.client.0.smithi162.stdout: "history": { 2024-09-16T08:03:23.042 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_created": 45, 2024-09-16T08:03:23.042 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_pool_created": 45, 2024-09-16T08:03:23.042 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 60, 2024-09-16T08:03:23.042 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_started": 58, 2024-09-16T08:03:23.042 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 46, 2024-09-16T08:03:23.042 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_clean": 45, 2024-09-16T08:03:23.042 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_split": 0, 2024-09-16T08:03:23.042 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_marked_full": 0, 2024-09-16T08:03:23.042 INFO:tasks.workunit.client.0.smithi162.stdout: "same_up_since": 58, 2024-09-16T08:03:23.042 INFO:tasks.workunit.client.0.smithi162.stdout: "same_interval_since": 58, 2024-09-16T08:03:23.042 INFO:tasks.workunit.client.0.smithi162.stdout: "same_primary_since": 45, 2024-09-16T08:03:23.042 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-16T08:03:23.042 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.043 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-16T08:03:23.043 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.043 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.043 INFO:tasks.workunit.client.0.smithi162.stdout: "prior_readable_until_ub": 0 2024-09-16T08:03:23.043 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.043 INFO:tasks.workunit.client.0.smithi162.stdout: "stats": { 2024-09-16T08:03:23.043 INFO:tasks.workunit.client.0.smithi162.stdout: "version": "53'101", 2024-09-16T08:03:23.043 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_seq": 129, 2024-09-16T08:03:23.043 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_epoch": 53, 2024-09-16T08:03:23.043 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "active+undersized+degraded", 2024-09-16T08:03:23.043 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fresh": "2024-09-16T08:02:59.217288+0000", 2024-09-16T08:03:23.043 INFO:tasks.workunit.client.0.smithi162.stdout: "last_change": "2024-09-16T08:02:29.869256+0000", 2024-09-16T08:03:23.044 INFO:tasks.workunit.client.0.smithi162.stdout: "last_active": "2024-09-16T08:02:59.217288+0000", 2024-09-16T08:03:23.044 INFO:tasks.workunit.client.0.smithi162.stdout: "last_peered": "2024-09-16T08:02:59.217288+0000", 2024-09-16T08:03:23.044 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean": "2024-09-16T08:02:28.075313+0000", 2024-09-16T08:03:23.044 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_active": "2024-09-16T08:02:29.869256+0000", 2024-09-16T08:03:23.044 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_peered": "2024-09-16T08:02:29.869256+0000", 2024-09-16T08:03:23.044 INFO:tasks.workunit.client.0.smithi162.stdout: "last_unstale": "2024-09-16T08:02:59.217288+0000", 2024-09-16T08:03:23.044 INFO:tasks.workunit.client.0.smithi162.stdout: "last_undegraded": "2024-09-16T08:02:29.525093+0000", 2024-09-16T08:03:23.044 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fullsized": "2024-09-16T08:02:29.524641+0000", 2024-09-16T08:03:23.044 INFO:tasks.workunit.client.0.smithi162.stdout: "mapping_epoch": 58, 2024-09-16T08:03:23.044 INFO:tasks.workunit.client.0.smithi162.stdout: "log_start": "0'0", 2024-09-16T08:03:23.044 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_start": "0'0", 2024-09-16T08:03:23.044 INFO:tasks.workunit.client.0.smithi162.stdout: "created": 45, 2024-09-16T08:03:23.044 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 46, 2024-09-16T08:03:23.044 INFO:tasks.workunit.client.0.smithi162.stdout: "parent": "0.0", 2024-09-16T08:03:23.045 INFO:tasks.workunit.client.0.smithi162.stdout: "parent_split_bits": 0, 2024-09-16T08:03:23.045 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-16T08:03:23.045 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.045 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-16T08:03:23.045 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.045 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.045 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_scrubbed": 0, 2024-09-16T08:03:23.045 INFO:tasks.workunit.client.0.smithi162.stdout: "log_size": 101, 2024-09-16T08:03:23.045 INFO:tasks.workunit.client.0.smithi162.stdout: "log_dups_size": 0, 2024-09-16T08:03:23.045 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_size": 101, 2024-09-16T08:03:23.045 INFO:tasks.workunit.client.0.smithi162.stdout: "stats_invalid": false, 2024-09-16T08:03:23.045 INFO:tasks.workunit.client.0.smithi162.stdout: "dirty_stats_invalid": false, 2024-09-16T08:03:23.045 INFO:tasks.workunit.client.0.smithi162.stdout: "omap_stats_invalid": false, 2024-09-16T08:03:23.046 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_stats_invalid": false, 2024-09-16T08:03:23.046 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_bytes_stats_invalid": false, 2024-09-16T08:03:23.046 INFO:tasks.workunit.client.0.smithi162.stdout: "pin_stats_invalid": false, 2024-09-16T08:03:23.046 INFO:tasks.workunit.client.0.smithi162.stdout: "manifest_stats_invalid": false, 2024-09-16T08:03:23.046 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrimq_len": 0, 2024-09-16T08:03:23.046 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_duration": 0, 2024-09-16T08:03:23.046 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-17T11:41:10.503212+0000", 2024-09-16T08:03:23.046 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_duration": 0, 2024-09-16T08:03:23.046 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_trimmed": 0, 2024-09-16T08:03:23.046 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrim_duration": 0, 2024-09-16T08:03:23.046 INFO:tasks.workunit.client.0.smithi162.stdout: "stat_sum": { 2024-09-16T08:03:23.046 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes": 413696, 2024-09-16T08:03:23.047 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects": 101, 2024-09-16T08:03:23.047 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_clones": 0, 2024-09-16T08:03:23.047 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_copies": 505, 2024-09-16T08:03:23.047 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T08:03:23.047 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing": 1, 2024-09-16T08:03:23.047 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_degraded": 101, 2024-09-16T08:03:23.047 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_misplaced": 0, 2024-09-16T08:03:23.047 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_unfound": 0, 2024-09-16T08:03:23.047 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_dirty": 101, 2024-09-16T08:03:23.047 INFO:tasks.workunit.client.0.smithi162.stdout: "num_whiteouts": 0, 2024-09-16T08:03:23.047 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read": 0, 2024-09-16T08:03:23.047 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read_kb": 0, 2024-09-16T08:03:23.047 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write": 101, 2024-09-16T08:03:23.048 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write_kb": 404, 2024-09-16T08:03:23.048 INFO:tasks.workunit.client.0.smithi162.stdout: "num_scrub_errors": 0, 2024-09-16T08:03:23.048 INFO:tasks.workunit.client.0.smithi162.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T08:03:23.048 INFO:tasks.workunit.client.0.smithi162.stdout: "num_deep_scrub_errors": 0, 2024-09-16T08:03:23.048 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_recovered": 0, 2024-09-16T08:03:23.048 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_recovered": 0, 2024-09-16T08:03:23.048 INFO:tasks.workunit.client.0.smithi162.stdout: "num_keys_recovered": 0, 2024-09-16T08:03:23.048 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_omap": 0, 2024-09-16T08:03:23.048 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T08:03:23.048 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T08:03:23.048 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush": 0, 2024-09-16T08:03:23.048 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_kb": 0, 2024-09-16T08:03:23.048 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict": 0, 2024-09-16T08:03:23.049 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_kb": 0, 2024-09-16T08:03:23.049 INFO:tasks.workunit.client.0.smithi162.stdout: "num_promote": 0, 2024-09-16T08:03:23.049 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_high": 0, 2024-09-16T08:03:23.049 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_low": 0, 2024-09-16T08:03:23.049 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_some": 0, 2024-09-16T08:03:23.049 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_full": 0, 2024-09-16T08:03:23.049 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_pinned": 0, 2024-09-16T08:03:23.049 INFO:tasks.workunit.client.0.smithi162.stdout: "num_legacy_snapsets": 0, 2024-09-16T08:03:23.049 INFO:tasks.workunit.client.0.smithi162.stdout: "num_large_omap_objects": 0, 2024-09-16T08:03:23.049 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_manifest": 0, 2024-09-16T08:03:23.049 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_bytes": 0, 2024-09-16T08:03:23.049 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_keys": 0, 2024-09-16T08:03:23.049 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_repaired": 0 2024-09-16T08:03:23.049 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.050 INFO:tasks.workunit.client.0.smithi162.stdout: "up": [ 2024-09-16T08:03:23.050 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-16T08:03:23.050 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-16T08:03:23.050 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-16T08:03:23.050 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-16T08:03:23.050 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-09-16T08:03:23.050 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:23.050 INFO:tasks.workunit.client.0.smithi162.stdout: "acting": [ 2024-09-16T08:03:23.050 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-16T08:03:23.050 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-16T08:03:23.050 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-16T08:03:23.050 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-16T08:03:23.050 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-09-16T08:03:23.051 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:23.051 INFO:tasks.workunit.client.0.smithi162.stdout: "avail_no_missing": [ 2024-09-16T08:03:23.051 INFO:tasks.workunit.client.0.smithi162.stdout: "3(0)", 2024-09-16T08:03:23.051 INFO:tasks.workunit.client.0.smithi162.stdout: "0(3)", 2024-09-16T08:03:23.051 INFO:tasks.workunit.client.0.smithi162.stdout: "1(1)", 2024-09-16T08:03:23.051 INFO:tasks.workunit.client.0.smithi162.stdout: "4(2)" 2024-09-16T08:03:23.051 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:23.051 INFO:tasks.workunit.client.0.smithi162.stdout: "object_location_counts": [ 2024-09-16T08:03:23.051 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T08:03:23.051 INFO:tasks.workunit.client.0.smithi162.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-16T08:03:23.051 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": 101 2024-09-16T08:03:23.051 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T08:03:23.051 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:23.052 INFO:tasks.workunit.client.0.smithi162.stdout: "blocked_by": [], 2024-09-16T08:03:23.052 INFO:tasks.workunit.client.0.smithi162.stdout: "up_primary": 3, 2024-09-16T08:03:23.052 INFO:tasks.workunit.client.0.smithi162.stdout: "acting_primary": 3, 2024-09-16T08:03:23.052 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [] 2024-09-16T08:03:23.052 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.052 INFO:tasks.workunit.client.0.smithi162.stdout: "empty": 0, 2024-09-16T08:03:23.052 INFO:tasks.workunit.client.0.smithi162.stdout: "dne": 0, 2024-09-16T08:03:23.052 INFO:tasks.workunit.client.0.smithi162.stdout: "incomplete": 0, 2024-09-16T08:03:23.052 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 60, 2024-09-16T08:03:23.052 INFO:tasks.workunit.client.0.smithi162.stdout: "hit_set_history": { 2024-09-16T08:03:23.052 INFO:tasks.workunit.client.0.smithi162.stdout: "current_last_update": "0'0", 2024-09-16T08:03:23.053 INFO:tasks.workunit.client.0.smithi162.stdout: "history": [] 2024-09-16T08:03:23.053 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T08:03:23.053 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.053 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T08:03:23.053 INFO:tasks.workunit.client.0.smithi162.stdout: "peer": "2(4)", 2024-09-16T08:03:23.053 INFO:tasks.workunit.client.0.smithi162.stdout: "pgid": "2.0s4", 2024-09-16T08:03:23.053 INFO:tasks.workunit.client.0.smithi162.stdout: "last_update": "53'101", 2024-09-16T08:03:23.053 INFO:tasks.workunit.client.0.smithi162.stdout: "last_complete": "53'8", 2024-09-16T08:03:23.053 INFO:tasks.workunit.client.0.smithi162.stdout: "log_tail": "0'0", 2024-09-16T08:03:23.053 INFO:tasks.workunit.client.0.smithi162.stdout: "last_user_version": 101, 2024-09-16T08:03:23.053 INFO:tasks.workunit.client.0.smithi162.stdout: "last_backfill": "MAX", 2024-09-16T08:03:23.053 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [], 2024-09-16T08:03:23.053 INFO:tasks.workunit.client.0.smithi162.stdout: "history": { 2024-09-16T08:03:23.053 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_created": 45, 2024-09-16T08:03:23.054 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_pool_created": 45, 2024-09-16T08:03:23.054 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 60, 2024-09-16T08:03:23.054 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_started": 58, 2024-09-16T08:03:23.054 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 46, 2024-09-16T08:03:23.054 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_clean": 45, 2024-09-16T08:03:23.054 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_split": 0, 2024-09-16T08:03:23.054 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_marked_full": 0, 2024-09-16T08:03:23.054 INFO:tasks.workunit.client.0.smithi162.stdout: "same_up_since": 58, 2024-09-16T08:03:23.054 INFO:tasks.workunit.client.0.smithi162.stdout: "same_interval_since": 58, 2024-09-16T08:03:23.054 INFO:tasks.workunit.client.0.smithi162.stdout: "same_primary_since": 45, 2024-09-16T08:03:23.054 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-16T08:03:23.054 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.054 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-16T08:03:23.054 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.055 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.055 INFO:tasks.workunit.client.0.smithi162.stdout: "prior_readable_until_ub": 0 2024-09-16T08:03:23.055 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.055 INFO:tasks.workunit.client.0.smithi162.stdout: "stats": { 2024-09-16T08:03:23.055 INFO:tasks.workunit.client.0.smithi162.stdout: "version": "53'101", 2024-09-16T08:03:23.055 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_seq": 134, 2024-09-16T08:03:23.055 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_epoch": 57, 2024-09-16T08:03:23.055 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "remapped+peering", 2024-09-16T08:03:23.055 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fresh": "2024-09-16T08:03:06.160537+0000", 2024-09-16T08:03:23.055 INFO:tasks.workunit.client.0.smithi162.stdout: "last_change": "2024-09-16T08:03:05.583241+0000", 2024-09-16T08:03:23.055 INFO:tasks.workunit.client.0.smithi162.stdout: "last_active": "2024-09-16T08:03:05.582929+0000", 2024-09-16T08:03:23.055 INFO:tasks.workunit.client.0.smithi162.stdout: "last_peered": "2024-09-16T08:03:05.297798+0000", 2024-09-16T08:03:23.055 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean": "2024-09-16T08:02:28.075313+0000", 2024-09-16T08:03:23.055 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_active": "2024-09-16T08:02:29.869256+0000", 2024-09-16T08:03:23.056 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_peered": "2024-09-16T08:02:29.869256+0000", 2024-09-16T08:03:23.056 INFO:tasks.workunit.client.0.smithi162.stdout: "last_unstale": "2024-09-16T08:03:06.160537+0000", 2024-09-16T08:03:23.056 INFO:tasks.workunit.client.0.smithi162.stdout: "last_undegraded": "2024-09-16T08:03:06.160537+0000", 2024-09-16T08:03:23.056 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fullsized": "2024-09-16T08:03:06.160537+0000", 2024-09-16T08:03:23.056 INFO:tasks.workunit.client.0.smithi162.stdout: "mapping_epoch": 58, 2024-09-16T08:03:23.056 INFO:tasks.workunit.client.0.smithi162.stdout: "log_start": "0'0", 2024-09-16T08:03:23.056 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_start": "0'0", 2024-09-16T08:03:23.056 INFO:tasks.workunit.client.0.smithi162.stdout: "created": 45, 2024-09-16T08:03:23.056 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 46, 2024-09-16T08:03:23.056 INFO:tasks.workunit.client.0.smithi162.stdout: "parent": "0.0", 2024-09-16T08:03:23.056 INFO:tasks.workunit.client.0.smithi162.stdout: "parent_split_bits": 0, 2024-09-16T08:03:23.056 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-16T08:03:23.057 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.057 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-16T08:03:23.057 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.057 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.057 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_scrubbed": 0, 2024-09-16T08:03:23.057 INFO:tasks.workunit.client.0.smithi162.stdout: "log_size": 101, 2024-09-16T08:03:23.057 INFO:tasks.workunit.client.0.smithi162.stdout: "log_dups_size": 0, 2024-09-16T08:03:23.057 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_size": 101, 2024-09-16T08:03:23.057 INFO:tasks.workunit.client.0.smithi162.stdout: "stats_invalid": false, 2024-09-16T08:03:23.057 INFO:tasks.workunit.client.0.smithi162.stdout: "dirty_stats_invalid": false, 2024-09-16T08:03:23.057 INFO:tasks.workunit.client.0.smithi162.stdout: "omap_stats_invalid": false, 2024-09-16T08:03:23.057 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_stats_invalid": false, 2024-09-16T08:03:23.057 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_bytes_stats_invalid": false, 2024-09-16T08:03:23.057 INFO:tasks.workunit.client.0.smithi162.stdout: "pin_stats_invalid": false, 2024-09-16T08:03:23.058 INFO:tasks.workunit.client.0.smithi162.stdout: "manifest_stats_invalid": false, 2024-09-16T08:03:23.058 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrimq_len": 0, 2024-09-16T08:03:23.058 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_duration": 0, 2024-09-16T08:03:23.058 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-17T19:01:54.469789+0000", 2024-09-16T08:03:23.058 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_duration": 0, 2024-09-16T08:03:23.058 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_trimmed": 0, 2024-09-16T08:03:23.058 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrim_duration": 0, 2024-09-16T08:03:23.058 INFO:tasks.workunit.client.0.smithi162.stdout: "stat_sum": { 2024-09-16T08:03:23.058 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes": 413696, 2024-09-16T08:03:23.058 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects": 101, 2024-09-16T08:03:23.058 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_clones": 0, 2024-09-16T08:03:23.058 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_copies": 505, 2024-09-16T08:03:23.058 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T08:03:23.058 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing": 1, 2024-09-16T08:03:23.059 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_degraded": 0, 2024-09-16T08:03:23.059 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_misplaced": 0, 2024-09-16T08:03:23.059 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_unfound": 0, 2024-09-16T08:03:23.059 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_dirty": 101, 2024-09-16T08:03:23.059 INFO:tasks.workunit.client.0.smithi162.stdout: "num_whiteouts": 0, 2024-09-16T08:03:23.059 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read": 0, 2024-09-16T08:03:23.059 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read_kb": 0, 2024-09-16T08:03:23.059 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write": 101, 2024-09-16T08:03:23.059 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write_kb": 404, 2024-09-16T08:03:23.059 INFO:tasks.workunit.client.0.smithi162.stdout: "num_scrub_errors": 0, 2024-09-16T08:03:23.059 INFO:tasks.workunit.client.0.smithi162.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T08:03:23.059 INFO:tasks.workunit.client.0.smithi162.stdout: "num_deep_scrub_errors": 0, 2024-09-16T08:03:23.059 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_recovered": 0, 2024-09-16T08:03:23.059 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_recovered": 0, 2024-09-16T08:03:23.060 INFO:tasks.workunit.client.0.smithi162.stdout: "num_keys_recovered": 0, 2024-09-16T08:03:23.060 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_omap": 0, 2024-09-16T08:03:23.060 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T08:03:23.060 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T08:03:23.060 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush": 0, 2024-09-16T08:03:23.060 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_kb": 0, 2024-09-16T08:03:23.060 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict": 0, 2024-09-16T08:03:23.060 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_kb": 0, 2024-09-16T08:03:23.060 INFO:tasks.workunit.client.0.smithi162.stdout: "num_promote": 0, 2024-09-16T08:03:23.060 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_high": 0, 2024-09-16T08:03:23.060 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_low": 0, 2024-09-16T08:03:23.060 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_some": 0, 2024-09-16T08:03:23.060 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_full": 0, 2024-09-16T08:03:23.060 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_pinned": 0, 2024-09-16T08:03:23.061 INFO:tasks.workunit.client.0.smithi162.stdout: "num_legacy_snapsets": 0, 2024-09-16T08:03:23.061 INFO:tasks.workunit.client.0.smithi162.stdout: "num_large_omap_objects": 0, 2024-09-16T08:03:23.061 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_manifest": 0, 2024-09-16T08:03:23.061 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_bytes": 0, 2024-09-16T08:03:23.061 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_keys": 0, 2024-09-16T08:03:23.061 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_repaired": 0 2024-09-16T08:03:23.061 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.061 INFO:tasks.workunit.client.0.smithi162.stdout: "up": [ 2024-09-16T08:03:23.061 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-16T08:03:23.061 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-16T08:03:23.061 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-16T08:03:23.061 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-16T08:03:23.061 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-09-16T08:03:23.062 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:23.062 INFO:tasks.workunit.client.0.smithi162.stdout: "acting": [ 2024-09-16T08:03:23.062 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-16T08:03:23.062 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-16T08:03:23.062 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-16T08:03:23.062 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-16T08:03:23.062 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-09-16T08:03:23.062 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:23.062 INFO:tasks.workunit.client.0.smithi162.stdout: "avail_no_missing": [], 2024-09-16T08:03:23.062 INFO:tasks.workunit.client.0.smithi162.stdout: "object_location_counts": [], 2024-09-16T08:03:23.062 INFO:tasks.workunit.client.0.smithi162.stdout: "blocked_by": [], 2024-09-16T08:03:23.062 INFO:tasks.workunit.client.0.smithi162.stdout: "up_primary": 3, 2024-09-16T08:03:23.062 INFO:tasks.workunit.client.0.smithi162.stdout: "acting_primary": 3, 2024-09-16T08:03:23.062 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [] 2024-09-16T08:03:23.063 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.063 INFO:tasks.workunit.client.0.smithi162.stdout: "empty": 0, 2024-09-16T08:03:23.063 INFO:tasks.workunit.client.0.smithi162.stdout: "dne": 0, 2024-09-16T08:03:23.063 INFO:tasks.workunit.client.0.smithi162.stdout: "incomplete": 0, 2024-09-16T08:03:23.063 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 60, 2024-09-16T08:03:23.063 INFO:tasks.workunit.client.0.smithi162.stdout: "hit_set_history": { 2024-09-16T08:03:23.063 INFO:tasks.workunit.client.0.smithi162.stdout: "current_last_update": "0'0", 2024-09-16T08:03:23.063 INFO:tasks.workunit.client.0.smithi162.stdout: "history": [] 2024-09-16T08:03:23.063 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T08:03:23.063 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.063 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T08:03:23.063 INFO:tasks.workunit.client.0.smithi162.stdout: "peer": "4(2)", 2024-09-16T08:03:23.063 INFO:tasks.workunit.client.0.smithi162.stdout: "pgid": "2.0s2", 2024-09-16T08:03:23.064 INFO:tasks.workunit.client.0.smithi162.stdout: "last_update": "53'101", 2024-09-16T08:03:23.064 INFO:tasks.workunit.client.0.smithi162.stdout: "last_complete": "53'101", 2024-09-16T08:03:23.064 INFO:tasks.workunit.client.0.smithi162.stdout: "log_tail": "0'0", 2024-09-16T08:03:23.064 INFO:tasks.workunit.client.0.smithi162.stdout: "last_user_version": 101, 2024-09-16T08:03:23.064 INFO:tasks.workunit.client.0.smithi162.stdout: "last_backfill": "MAX", 2024-09-16T08:03:23.064 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [], 2024-09-16T08:03:23.064 INFO:tasks.workunit.client.0.smithi162.stdout: "history": { 2024-09-16T08:03:23.064 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_created": 45, 2024-09-16T08:03:23.064 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_pool_created": 45, 2024-09-16T08:03:23.064 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 60, 2024-09-16T08:03:23.064 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_started": 58, 2024-09-16T08:03:23.064 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 46, 2024-09-16T08:03:23.065 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_clean": 45, 2024-09-16T08:03:23.065 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_split": 0, 2024-09-16T08:03:23.065 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_marked_full": 0, 2024-09-16T08:03:23.065 INFO:tasks.workunit.client.0.smithi162.stdout: "same_up_since": 58, 2024-09-16T08:03:23.065 INFO:tasks.workunit.client.0.smithi162.stdout: "same_interval_since": 58, 2024-09-16T08:03:23.065 INFO:tasks.workunit.client.0.smithi162.stdout: "same_primary_since": 45, 2024-09-16T08:03:23.065 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-16T08:03:23.065 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.065 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-16T08:03:23.065 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.065 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.065 INFO:tasks.workunit.client.0.smithi162.stdout: "prior_readable_until_ub": 0 2024-09-16T08:03:23.065 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.065 INFO:tasks.workunit.client.0.smithi162.stdout: "stats": { 2024-09-16T08:03:23.066 INFO:tasks.workunit.client.0.smithi162.stdout: "version": "53'101", 2024-09-16T08:03:23.066 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_seq": 129, 2024-09-16T08:03:23.066 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_epoch": 53, 2024-09-16T08:03:23.066 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "active+undersized+degraded", 2024-09-16T08:03:23.066 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fresh": "2024-09-16T08:02:59.217288+0000", 2024-09-16T08:03:23.066 INFO:tasks.workunit.client.0.smithi162.stdout: "last_change": "2024-09-16T08:02:29.869256+0000", 2024-09-16T08:03:23.066 INFO:tasks.workunit.client.0.smithi162.stdout: "last_active": "2024-09-16T08:02:59.217288+0000", 2024-09-16T08:03:23.066 INFO:tasks.workunit.client.0.smithi162.stdout: "last_peered": "2024-09-16T08:02:59.217288+0000", 2024-09-16T08:03:23.066 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean": "2024-09-16T08:02:28.075313+0000", 2024-09-16T08:03:23.066 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_active": "2024-09-16T08:02:29.869256+0000", 2024-09-16T08:03:23.066 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_peered": "2024-09-16T08:02:29.869256+0000", 2024-09-16T08:03:23.066 INFO:tasks.workunit.client.0.smithi162.stdout: "last_unstale": "2024-09-16T08:02:59.217288+0000", 2024-09-16T08:03:23.066 INFO:tasks.workunit.client.0.smithi162.stdout: "last_undegraded": "2024-09-16T08:02:29.525093+0000", 2024-09-16T08:03:23.067 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fullsized": "2024-09-16T08:02:29.524641+0000", 2024-09-16T08:03:23.067 INFO:tasks.workunit.client.0.smithi162.stdout: "mapping_epoch": 58, 2024-09-16T08:03:23.067 INFO:tasks.workunit.client.0.smithi162.stdout: "log_start": "0'0", 2024-09-16T08:03:23.067 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_start": "0'0", 2024-09-16T08:03:23.067 INFO:tasks.workunit.client.0.smithi162.stdout: "created": 45, 2024-09-16T08:03:23.067 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 46, 2024-09-16T08:03:23.067 INFO:tasks.workunit.client.0.smithi162.stdout: "parent": "0.0", 2024-09-16T08:03:23.067 INFO:tasks.workunit.client.0.smithi162.stdout: "parent_split_bits": 0, 2024-09-16T08:03:23.067 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-16T08:03:23.067 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.067 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-16T08:03:23.067 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.067 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-16T08:02:19.028848+0000", 2024-09-16T08:03:23.067 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_scrubbed": 0, 2024-09-16T08:03:23.068 INFO:tasks.workunit.client.0.smithi162.stdout: "log_size": 101, 2024-09-16T08:03:23.068 INFO:tasks.workunit.client.0.smithi162.stdout: "log_dups_size": 0, 2024-09-16T08:03:23.068 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_size": 101, 2024-09-16T08:03:23.068 INFO:tasks.workunit.client.0.smithi162.stdout: "stats_invalid": false, 2024-09-16T08:03:23.068 INFO:tasks.workunit.client.0.smithi162.stdout: "dirty_stats_invalid": false, 2024-09-16T08:03:23.068 INFO:tasks.workunit.client.0.smithi162.stdout: "omap_stats_invalid": false, 2024-09-16T08:03:23.068 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_stats_invalid": false, 2024-09-16T08:03:23.068 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_bytes_stats_invalid": false, 2024-09-16T08:03:23.068 INFO:tasks.workunit.client.0.smithi162.stdout: "pin_stats_invalid": false, 2024-09-16T08:03:23.068 INFO:tasks.workunit.client.0.smithi162.stdout: "manifest_stats_invalid": false, 2024-09-16T08:03:23.068 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrimq_len": 0, 2024-09-16T08:03:23.068 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_duration": 0, 2024-09-16T08:03:23.068 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-17T11:41:10.503212+0000", 2024-09-16T08:03:23.068 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_duration": 0, 2024-09-16T08:03:23.068 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_trimmed": 0, 2024-09-16T08:03:23.069 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrim_duration": 0, 2024-09-16T08:03:23.069 INFO:tasks.workunit.client.0.smithi162.stdout: "stat_sum": { 2024-09-16T08:03:23.069 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes": 413696, 2024-09-16T08:03:23.069 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects": 101, 2024-09-16T08:03:23.069 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_clones": 0, 2024-09-16T08:03:23.069 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_copies": 505, 2024-09-16T08:03:23.069 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T08:03:23.069 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing": 0, 2024-09-16T08:03:23.069 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_degraded": 101, 2024-09-16T08:03:23.069 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_misplaced": 0, 2024-09-16T08:03:23.069 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_unfound": 0, 2024-09-16T08:03:23.069 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_dirty": 101, 2024-09-16T08:03:23.069 INFO:tasks.workunit.client.0.smithi162.stdout: "num_whiteouts": 0, 2024-09-16T08:03:23.070 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read": 0, 2024-09-16T08:03:23.070 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read_kb": 0, 2024-09-16T08:03:23.070 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write": 101, 2024-09-16T08:03:23.070 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write_kb": 404, 2024-09-16T08:03:23.070 INFO:tasks.workunit.client.0.smithi162.stdout: "num_scrub_errors": 0, 2024-09-16T08:03:23.070 INFO:tasks.workunit.client.0.smithi162.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T08:03:23.070 INFO:tasks.workunit.client.0.smithi162.stdout: "num_deep_scrub_errors": 0, 2024-09-16T08:03:23.070 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_recovered": 0, 2024-09-16T08:03:23.070 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_recovered": 0, 2024-09-16T08:03:23.070 INFO:tasks.workunit.client.0.smithi162.stdout: "num_keys_recovered": 0, 2024-09-16T08:03:23.070 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_omap": 0, 2024-09-16T08:03:23.070 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T08:03:23.070 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T08:03:23.070 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush": 0, 2024-09-16T08:03:23.071 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_kb": 0, 2024-09-16T08:03:23.071 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict": 0, 2024-09-16T08:03:23.071 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_kb": 0, 2024-09-16T08:03:23.071 INFO:tasks.workunit.client.0.smithi162.stdout: "num_promote": 0, 2024-09-16T08:03:23.071 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_high": 0, 2024-09-16T08:03:23.071 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_low": 0, 2024-09-16T08:03:23.071 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_some": 0, 2024-09-16T08:03:23.071 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_full": 0, 2024-09-16T08:03:23.071 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_pinned": 0, 2024-09-16T08:03:23.071 INFO:tasks.workunit.client.0.smithi162.stdout: "num_legacy_snapsets": 0, 2024-09-16T08:03:23.071 INFO:tasks.workunit.client.0.smithi162.stdout: "num_large_omap_objects": 0, 2024-09-16T08:03:23.071 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_manifest": 0, 2024-09-16T08:03:23.071 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_bytes": 0, 2024-09-16T08:03:23.072 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_keys": 0, 2024-09-16T08:03:23.072 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_repaired": 0 2024-09-16T08:03:23.072 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.072 INFO:tasks.workunit.client.0.smithi162.stdout: "up": [ 2024-09-16T08:03:23.072 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-16T08:03:23.072 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-16T08:03:23.072 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-16T08:03:23.072 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-16T08:03:23.072 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-09-16T08:03:23.072 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:23.072 INFO:tasks.workunit.client.0.smithi162.stdout: "acting": [ 2024-09-16T08:03:23.072 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-16T08:03:23.072 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-16T08:03:23.073 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-16T08:03:23.073 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-16T08:03:23.073 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-09-16T08:03:23.073 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:23.073 INFO:tasks.workunit.client.0.smithi162.stdout: "avail_no_missing": [ 2024-09-16T08:03:23.073 INFO:tasks.workunit.client.0.smithi162.stdout: "3(0)", 2024-09-16T08:03:23.073 INFO:tasks.workunit.client.0.smithi162.stdout: "0(3)", 2024-09-16T08:03:23.073 INFO:tasks.workunit.client.0.smithi162.stdout: "1(1)", 2024-09-16T08:03:23.073 INFO:tasks.workunit.client.0.smithi162.stdout: "4(2)" 2024-09-16T08:03:23.073 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:23.073 INFO:tasks.workunit.client.0.smithi162.stdout: "object_location_counts": [ 2024-09-16T08:03:23.073 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T08:03:23.073 INFO:tasks.workunit.client.0.smithi162.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-16T08:03:23.073 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": 101 2024-09-16T08:03:23.074 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T08:03:23.074 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:23.074 INFO:tasks.workunit.client.0.smithi162.stdout: "blocked_by": [], 2024-09-16T08:03:23.074 INFO:tasks.workunit.client.0.smithi162.stdout: "up_primary": 3, 2024-09-16T08:03:23.074 INFO:tasks.workunit.client.0.smithi162.stdout: "acting_primary": 3, 2024-09-16T08:03:23.074 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [] 2024-09-16T08:03:23.074 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.074 INFO:tasks.workunit.client.0.smithi162.stdout: "empty": 0, 2024-09-16T08:03:23.074 INFO:tasks.workunit.client.0.smithi162.stdout: "dne": 0, 2024-09-16T08:03:23.074 INFO:tasks.workunit.client.0.smithi162.stdout: "incomplete": 0, 2024-09-16T08:03:23.074 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 60, 2024-09-16T08:03:23.074 INFO:tasks.workunit.client.0.smithi162.stdout: "hit_set_history": { 2024-09-16T08:03:23.074 INFO:tasks.workunit.client.0.smithi162.stdout: "current_last_update": "0'0", 2024-09-16T08:03:23.074 INFO:tasks.workunit.client.0.smithi162.stdout: "history": [] 2024-09-16T08:03:23.075 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T08:03:23.075 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T08:03:23.075 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:23.075 INFO:tasks.workunit.client.0.smithi162.stdout: "recovery_state": [ 2024-09-16T08:03:23.075 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T08:03:23.075 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "Started/Primary/Active", 2024-09-16T08:03:23.075 INFO:tasks.workunit.client.0.smithi162.stdout: "enter_time": "2024-09-16T08:03:07.916377+0000", 2024-09-16T08:03:23.075 INFO:tasks.workunit.client.0.smithi162.stdout: "might_have_unfound": [ 2024-09-16T08:03:23.075 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T08:03:23.075 INFO:tasks.workunit.client.0.smithi162.stdout: "osd": "0(3)", 2024-09-16T08:03:23.075 INFO:tasks.workunit.client.0.smithi162.stdout: "status": "already probed" 2024-09-16T08:03:23.075 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.075 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T08:03:23.076 INFO:tasks.workunit.client.0.smithi162.stdout: "osd": "1(1)", 2024-09-16T08:03:23.076 INFO:tasks.workunit.client.0.smithi162.stdout: "status": "already probed" 2024-09-16T08:03:23.076 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.076 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T08:03:23.076 INFO:tasks.workunit.client.0.smithi162.stdout: "osd": "2(4)", 2024-09-16T08:03:23.076 INFO:tasks.workunit.client.0.smithi162.stdout: "status": "already probed" 2024-09-16T08:03:23.076 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.076 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T08:03:23.076 INFO:tasks.workunit.client.0.smithi162.stdout: "osd": "4(2)", 2024-09-16T08:03:23.076 INFO:tasks.workunit.client.0.smithi162.stdout: "status": "already probed" 2024-09-16T08:03:23.076 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T08:03:23.076 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:23.076 INFO:tasks.workunit.client.0.smithi162.stdout: "recovery_progress": { 2024-09-16T08:03:23.076 INFO:tasks.workunit.client.0.smithi162.stdout: "backfill_targets": [], 2024-09-16T08:03:23.077 INFO:tasks.workunit.client.0.smithi162.stdout: "waiting_on_backfill": [], 2024-09-16T08:03:23.077 INFO:tasks.workunit.client.0.smithi162.stdout: "last_backfill_started": "MIN", 2024-09-16T08:03:23.077 INFO:tasks.workunit.client.0.smithi162.stdout: "backfill_info": { 2024-09-16T08:03:23.077 INFO:tasks.workunit.client.0.smithi162.stdout: "begin": "MIN", 2024-09-16T08:03:23.077 INFO:tasks.workunit.client.0.smithi162.stdout: "end": "MIN", 2024-09-16T08:03:23.077 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": [] 2024-09-16T08:03:23.077 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.077 INFO:tasks.workunit.client.0.smithi162.stdout: "peer_backfill_info": [], 2024-09-16T08:03:23.077 INFO:tasks.workunit.client.0.smithi162.stdout: "backfills_in_flight": [], 2024-09-16T08:03:23.077 INFO:tasks.workunit.client.0.smithi162.stdout: "recovering": [], 2024-09-16T08:03:23.077 INFO:tasks.workunit.client.0.smithi162.stdout: "pg_backend": { 2024-09-16T08:03:23.077 INFO:tasks.workunit.client.0.smithi162.stdout: "recovery_ops": [], 2024-09-16T08:03:23.078 INFO:tasks.workunit.client.0.smithi162.stdout: "read_ops": [] 2024-09-16T08:03:23.078 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T08:03:23.078 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T08:03:23.078 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.078 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-16T08:03:23.078 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "Started", 2024-09-16T08:03:23.078 INFO:tasks.workunit.client.0.smithi162.stdout: "enter_time": "2024-09-16T08:03:07.089111+0000" 2024-09-16T08:03:23.078 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-16T08:03:23.078 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-16T08:03:23.078 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubber": { 2024-09-16T08:03:23.078 INFO:tasks.workunit.client.0.smithi162.stdout: "active": false, 2024-09-16T08:03:23.078 INFO:tasks.workunit.client.0.smithi162.stdout: "must_scrub": false, 2024-09-16T08:03:23.078 INFO:tasks.workunit.client.0.smithi162.stdout: "must_deep_scrub": false, 2024-09-16T08:03:23.079 INFO:tasks.workunit.client.0.smithi162.stdout: "must_repair": false, 2024-09-16T08:03:23.079 INFO:tasks.workunit.client.0.smithi162.stdout: "need_auto": false, 2024-09-16T08:03:23.079 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_reg_stamp": "2024-09-17T18:39:40.559784+0000", 2024-09-16T08:03:23.079 INFO:tasks.workunit.client.0.smithi162.stdout: "schedule": "scrub scheduled @ 2024-09-17T18:39:40.559" 2024-09-16T08:03:23.079 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-16T08:03:23.079 INFO:tasks.workunit.client.0.smithi162.stdout: "agent_state": {} 2024-09-16T08:03:23.079 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-09-16T08:03:23.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:664: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-16T08:03:23.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:664: TEST_ec_recovery_unfound: grep -q obj75 2024-09-16T08:03:23.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-16T08:03:23.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: jq .available_might_have_unfound 2024-09-16T08:03:23.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: check=true 2024-09-16T08:03:23.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:667: TEST_ec_recovery_unfound: test true == true 2024-09-16T08:03:23.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:669: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-16T08:03:23.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:669: TEST_ec_recovery_unfound: jq '.might_have_unfound | length' 2024-09-16T08:03:23.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:669: TEST_ec_recovery_unfound: check=0 2024-09-16T08:03:23.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:670: TEST_ec_recovery_unfound: test 0 == 0 2024-09-16T08:03:23.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:673: TEST_ec_recovery_unfound: timeout 5 rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-09-16T08:03:28.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:674: TEST_ec_recovery_unfound: test 124 = 124 2024-09-16T08:03:28.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:676: TEST_ec_recovery_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-09-16T08:03:28.824 INFO:tasks.workunit.client.0.smithi162.stderr:pg has no unfound objects 2024-09-16T08:03:28.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:678: TEST_ec_recovery_unfound: wait_for_clean 2024-09-16T08:03:28.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:03:28.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:03:28.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:03:28.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:03:28.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:03:28.837 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:03:28.837 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:03:28.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:03:28.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:03:28.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:03:29.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:03:29.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:03:29.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:03:29.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:03:29.058 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:03:29.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:03:29.538 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:03:29.538 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:03:29.539 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T08:03:29.539 INFO:tasks.workunit.client.0.smithi162.stderr:4' 2024-09-16T08:03:29.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:03:29.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:03:29.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:03:29.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803838 2024-09-16T08:03:29.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803838 2024-09-16T08:03:29.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803838' 2024-09-16T08:03:29.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:03:29.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:03:29.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607607 2024-09-16T08:03:29.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607607 2024-09-16T08:03:29.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803838 1-51539607607' 2024-09-16T08:03:29.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:03:29.852 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:03:30.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=236223201288 2024-09-16T08:03:30.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 236223201288 2024-09-16T08:03:30.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803838 1-51539607607 2-236223201288' 2024-09-16T08:03:30.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:03:30.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:03:30.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=120259084331 2024-09-16T08:03:30.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 120259084331 2024-09-16T08:03:30.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803838 1-51539607607 2-236223201288 3-120259084331' 2024-09-16T08:03:30.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:03:30.155 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T08:03:30.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=171798691876 2024-09-16T08:03:30.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 171798691876 2024-09-16T08:03:30.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803838 1-51539607607 2-236223201288 3-120259084331 4-171798691876' 2024-09-16T08:03:30.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:03:30.313 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803838 2024-09-16T08:03:30.314 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:03:30.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:03:30.317 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803838 2024-09-16T08:03:30.317 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:03:30.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803838 2024-09-16T08:03:30.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803838' 2024-09-16T08:03:30.318 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803838 2024-09-16T08:03:30.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:03:30.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803838 -lt 25769803838 2024-09-16T08:03:30.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:03:30.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607607 2024-09-16T08:03:30.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:03:30.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:03:30.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607607 2024-09-16T08:03:30.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:03:30.869 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 51539607607 2024-09-16T08:03:30.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607607 2024-09-16T08:03:30.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607607' 2024-09-16T08:03:30.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:03:31.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607607 -lt 51539607607 2024-09-16T08:03:31.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:03:31.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-236223201288 2024-09-16T08:03:31.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:03:31.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:03:31.294 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-236223201288 2024-09-16T08:03:31.294 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:03:31.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=236223201288 2024-09-16T08:03:31.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 236223201288' 2024-09-16T08:03:31.296 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 236223201288 2024-09-16T08:03:31.297 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:03:31.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 236223201286 -lt 236223201288 2024-09-16T08:03:31.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T08:03:32.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T08:03:32.747 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:03:33.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 236223201289 -lt 236223201288 2024-09-16T08:03:33.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:03:33.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-120259084331 2024-09-16T08:03:33.160 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:03:33.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:03:33.163 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:03:33.163 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-120259084331 2024-09-16T08:03:33.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=120259084331 2024-09-16T08:03:33.164 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 120259084331 2024-09-16T08:03:33.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 120259084331' 2024-09-16T08:03:33.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:03:33.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 120259084331 -lt 120259084331 2024-09-16T08:03:33.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:03:33.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-171798691876 2024-09-16T08:03:33.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:03:33.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T08:03:33.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:03:33.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-171798691876 2024-09-16T08:03:33.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=171798691876 2024-09-16T08:03:33.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 171798691876' 2024-09-16T08:03:33.601 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 171798691876 2024-09-16T08:03:33.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T08:03:34.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 171798691876 -lt 171798691876 2024-09-16T08:03:34.027 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:03:34.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:03:34.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:03:34.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:03:34.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:03:34.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:03:34.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:03:34.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:03:34.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:03:34.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:03:34.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:03:35.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:03:35.027 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:03:35.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:03:35.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:03:35.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:03:35.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:03:35.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:03:35.553 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: seq 1 100 2024-09-16T08:03:35.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:35.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj1 = obj75 ']' 2024-09-16T08:03:35.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2024-09-16T08:03:35.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:35.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:35.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj2 = obj75 ']' 2024-09-16T08:03:35.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2024-09-16T08:03:35.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:35.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:35.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj3 = obj75 ']' 2024-09-16T08:03:35.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2024-09-16T08:03:35.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:35.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:35.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj4 = obj75 ']' 2024-09-16T08:03:35.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2024-09-16T08:03:35.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:35.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:35.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj5 = obj75 ']' 2024-09-16T08:03:35.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2024-09-16T08:03:35.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:35.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:35.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj6 = obj75 ']' 2024-09-16T08:03:35.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2024-09-16T08:03:35.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:35.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:35.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj7 = obj75 ']' 2024-09-16T08:03:35.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2024-09-16T08:03:35.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:35.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:35.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj8 = obj75 ']' 2024-09-16T08:03:35.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2024-09-16T08:03:36.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:36.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:36.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj9 = obj75 ']' 2024-09-16T08:03:36.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2024-09-16T08:03:36.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:36.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:36.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj10 = obj75 ']' 2024-09-16T08:03:36.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2024-09-16T08:03:36.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:36.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:36.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj11 = obj75 ']' 2024-09-16T08:03:36.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2024-09-16T08:03:36.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:36.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:36.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj12 = obj75 ']' 2024-09-16T08:03:36.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2024-09-16T08:03:36.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:36.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:36.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj13 = obj75 ']' 2024-09-16T08:03:36.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2024-09-16T08:03:36.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:36.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:36.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj14 = obj75 ']' 2024-09-16T08:03:36.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2024-09-16T08:03:36.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:36.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:36.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj15 = obj75 ']' 2024-09-16T08:03:36.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2024-09-16T08:03:36.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:36.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:36.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj16 = obj75 ']' 2024-09-16T08:03:36.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2024-09-16T08:03:36.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:36.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:36.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj17 = obj75 ']' 2024-09-16T08:03:36.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2024-09-16T08:03:36.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:36.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:36.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj18 = obj75 ']' 2024-09-16T08:03:36.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2024-09-16T08:03:36.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:36.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:36.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj19 = obj75 ']' 2024-09-16T08:03:36.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2024-09-16T08:03:36.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:36.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:36.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj20 = obj75 ']' 2024-09-16T08:03:36.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2024-09-16T08:03:36.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:36.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:36.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj21 = obj75 ']' 2024-09-16T08:03:36.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2024-09-16T08:03:36.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:36.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:36.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj22 = obj75 ']' 2024-09-16T08:03:36.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2024-09-16T08:03:36.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:36.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:36.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj23 = obj75 ']' 2024-09-16T08:03:36.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2024-09-16T08:03:36.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:36.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:36.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj24 = obj75 ']' 2024-09-16T08:03:36.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2024-09-16T08:03:36.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:36.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:36.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj25 = obj75 ']' 2024-09-16T08:03:36.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2024-09-16T08:03:36.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:36.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:36.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj26 = obj75 ']' 2024-09-16T08:03:36.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2024-09-16T08:03:36.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:36.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:36.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj27 = obj75 ']' 2024-09-16T08:03:36.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2024-09-16T08:03:36.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:36.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:36.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj28 = obj75 ']' 2024-09-16T08:03:36.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2024-09-16T08:03:37.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:37.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:37.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj29 = obj75 ']' 2024-09-16T08:03:37.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2024-09-16T08:03:37.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:37.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:37.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj30 = obj75 ']' 2024-09-16T08:03:37.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2024-09-16T08:03:37.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:37.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:37.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj31 = obj75 ']' 2024-09-16T08:03:37.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2024-09-16T08:03:37.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:37.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:37.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj32 = obj75 ']' 2024-09-16T08:03:37.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2024-09-16T08:03:37.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:37.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:37.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj33 = obj75 ']' 2024-09-16T08:03:37.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2024-09-16T08:03:37.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:37.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:37.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj34 = obj75 ']' 2024-09-16T08:03:37.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2024-09-16T08:03:37.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:37.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:37.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj35 = obj75 ']' 2024-09-16T08:03:37.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2024-09-16T08:03:37.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:37.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:37.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj36 = obj75 ']' 2024-09-16T08:03:37.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2024-09-16T08:03:37.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:37.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:37.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj37 = obj75 ']' 2024-09-16T08:03:37.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2024-09-16T08:03:37.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:37.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:37.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj38 = obj75 ']' 2024-09-16T08:03:37.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2024-09-16T08:03:37.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:37.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:37.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj39 = obj75 ']' 2024-09-16T08:03:37.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2024-09-16T08:03:37.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:37.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:37.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj40 = obj75 ']' 2024-09-16T08:03:37.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2024-09-16T08:03:37.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:37.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:37.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj41 = obj75 ']' 2024-09-16T08:03:37.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2024-09-16T08:03:37.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:37.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:37.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj42 = obj75 ']' 2024-09-16T08:03:37.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2024-09-16T08:03:37.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:37.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:37.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj43 = obj75 ']' 2024-09-16T08:03:37.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2024-09-16T08:03:37.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:37.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:37.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj44 = obj75 ']' 2024-09-16T08:03:37.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2024-09-16T08:03:37.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:37.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:37.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj45 = obj75 ']' 2024-09-16T08:03:37.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2024-09-16T08:03:37.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:37.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:37.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj46 = obj75 ']' 2024-09-16T08:03:37.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2024-09-16T08:03:37.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:37.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:37.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj47 = obj75 ']' 2024-09-16T08:03:37.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2024-09-16T08:03:37.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:37.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:37.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj48 = obj75 ']' 2024-09-16T08:03:37.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2024-09-16T08:03:38.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:38.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:38.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj49 = obj75 ']' 2024-09-16T08:03:38.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2024-09-16T08:03:38.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:38.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:38.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj50 = obj75 ']' 2024-09-16T08:03:38.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2024-09-16T08:03:38.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:38.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:38.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj51 = obj75 ']' 2024-09-16T08:03:38.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2024-09-16T08:03:38.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:38.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:38.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj52 = obj75 ']' 2024-09-16T08:03:38.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2024-09-16T08:03:38.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:38.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:38.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj53 = obj75 ']' 2024-09-16T08:03:38.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2024-09-16T08:03:38.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:38.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:38.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj54 = obj75 ']' 2024-09-16T08:03:38.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2024-09-16T08:03:38.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:38.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:38.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj55 = obj75 ']' 2024-09-16T08:03:38.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2024-09-16T08:03:38.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:38.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:38.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj56 = obj75 ']' 2024-09-16T08:03:38.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2024-09-16T08:03:38.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:38.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:38.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj57 = obj75 ']' 2024-09-16T08:03:38.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2024-09-16T08:03:38.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:38.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:38.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj58 = obj75 ']' 2024-09-16T08:03:38.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2024-09-16T08:03:38.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:38.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:38.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj59 = obj75 ']' 2024-09-16T08:03:38.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2024-09-16T08:03:38.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:38.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:38.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj60 = obj75 ']' 2024-09-16T08:03:38.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2024-09-16T08:03:38.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:38.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:38.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj61 = obj75 ']' 2024-09-16T08:03:38.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2024-09-16T08:03:38.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:38.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:38.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj62 = obj75 ']' 2024-09-16T08:03:38.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2024-09-16T08:03:38.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:38.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:38.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj63 = obj75 ']' 2024-09-16T08:03:38.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2024-09-16T08:03:38.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:38.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:38.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj64 = obj75 ']' 2024-09-16T08:03:38.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2024-09-16T08:03:38.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:38.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:38.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj65 = obj75 ']' 2024-09-16T08:03:38.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2024-09-16T08:03:38.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:38.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:38.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj66 = obj75 ']' 2024-09-16T08:03:38.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2024-09-16T08:03:38.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:38.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:38.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj67 = obj75 ']' 2024-09-16T08:03:38.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2024-09-16T08:03:38.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:38.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:38.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj68 = obj75 ']' 2024-09-16T08:03:38.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2024-09-16T08:03:39.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:39.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:39.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj69 = obj75 ']' 2024-09-16T08:03:39.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2024-09-16T08:03:39.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:39.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:39.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj70 = obj75 ']' 2024-09-16T08:03:39.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2024-09-16T08:03:39.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:39.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:39.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj71 = obj75 ']' 2024-09-16T08:03:39.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2024-09-16T08:03:39.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:39.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:39.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj72 = obj75 ']' 2024-09-16T08:03:39.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2024-09-16T08:03:39.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:39.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:39.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj73 = obj75 ']' 2024-09-16T08:03:39.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2024-09-16T08:03:39.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:39.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:39.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj74 = obj75 ']' 2024-09-16T08:03:39.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2024-09-16T08:03:39.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:39.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:39.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj75 = obj75 ']' 2024-09-16T08:03:39.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:684: TEST_ec_recovery_unfound: rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-09-16T08:03:39.364 INFO:tasks.workunit.client.0.smithi162.stderr:error getting pool-jerasure/obj75: (2) No such file or directory 2024-09-16T08:03:39.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:39.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj76 = obj75 ']' 2024-09-16T08:03:39.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2024-09-16T08:03:39.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:39.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:39.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj77 = obj75 ']' 2024-09-16T08:03:39.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2024-09-16T08:03:39.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:39.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:39.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj78 = obj75 ']' 2024-09-16T08:03:39.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2024-09-16T08:03:39.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:39.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:39.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj79 = obj75 ']' 2024-09-16T08:03:39.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2024-09-16T08:03:39.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:39.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:39.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj80 = obj75 ']' 2024-09-16T08:03:39.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2024-09-16T08:03:39.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:39.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:39.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj81 = obj75 ']' 2024-09-16T08:03:39.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2024-09-16T08:03:39.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:39.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:39.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj82 = obj75 ']' 2024-09-16T08:03:39.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2024-09-16T08:03:39.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:39.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:39.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj83 = obj75 ']' 2024-09-16T08:03:39.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2024-09-16T08:03:39.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:39.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:39.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj84 = obj75 ']' 2024-09-16T08:03:39.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2024-09-16T08:03:39.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:39.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:39.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj85 = obj75 ']' 2024-09-16T08:03:39.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2024-09-16T08:03:39.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:39.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:39.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj86 = obj75 ']' 2024-09-16T08:03:39.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2024-09-16T08:03:39.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:39.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:39.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj87 = obj75 ']' 2024-09-16T08:03:39.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2024-09-16T08:03:39.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:39.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:39.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj88 = obj75 ']' 2024-09-16T08:03:39.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2024-09-16T08:03:40.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:40.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:40.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj89 = obj75 ']' 2024-09-16T08:03:40.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2024-09-16T08:03:40.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:40.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:40.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj90 = obj75 ']' 2024-09-16T08:03:40.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2024-09-16T08:03:40.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:40.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:40.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj91 = obj75 ']' 2024-09-16T08:03:40.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2024-09-16T08:03:40.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:40.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:40.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj92 = obj75 ']' 2024-09-16T08:03:40.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2024-09-16T08:03:40.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:40.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:40.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj93 = obj75 ']' 2024-09-16T08:03:40.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2024-09-16T08:03:40.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:40.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:40.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj94 = obj75 ']' 2024-09-16T08:03:40.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2024-09-16T08:03:40.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:40.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:40.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj95 = obj75 ']' 2024-09-16T08:03:40.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2024-09-16T08:03:40.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:40.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:40.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj96 = obj75 ']' 2024-09-16T08:03:40.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2024-09-16T08:03:40.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:40.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:40.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj97 = obj75 ']' 2024-09-16T08:03:40.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2024-09-16T08:03:40.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:40.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:40.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj98 = obj75 ']' 2024-09-16T08:03:40.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2024-09-16T08:03:40.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:40.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:40.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj99 = obj75 ']' 2024-09-16T08:03:40.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2024-09-16T08:03:40.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:40.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-16T08:03:40.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj100 = obj75 ']' 2024-09-16T08:03:40.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2024-09-16T08:03:40.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:40.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:691: TEST_ec_recovery_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-16T08:03:40.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:693: TEST_ec_recovery_unfound: delete_erasure_coded_pool pool-jerasure 2024-09-16T08:03:40.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T08:03:40.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-16T08:03:41.325 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-16T08:03:41.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-16T08:03:42.066 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-16T08:03:42.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-16T08:03:42.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T08:03:42.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T08:03:42.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T08:03:42.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:03:42.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:03:42.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:03:42.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:03:42.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:03:42.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:03:42.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T08:03:42.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T08:03:42.397 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T08:03:42.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T08:03:42.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T08:03:42.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T08:03:42.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:03:42.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T08:03:42.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T08:03:42.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:03:42.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:03:42.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T08:03:42.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T08:03:42.446 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T08:03:42.446 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:03:42.446 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:03:42.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T08:03:42.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T08:03:42.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T08:03:42.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-16T08:03:42.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-16T08:03:42.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-16T08:03:42.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-16T08:03:42.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T08:03:42.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T08:03:42.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T08:03:42.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:03:42.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:03:42.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:03:42.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:03:42.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:03:42.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:03:42.454 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T08:03:42.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T08:03:42.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T08:03:42.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T08:03:42.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T08:03:42.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T08:03:42.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:03:42.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T08:03:42.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T08:03:42.462 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:03:42.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:03:42.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T08:03:42.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T08:03:42.468 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T08:03:42.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:03:42.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:03:42.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T08:03:42.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T08:03:42.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T08:03:42.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-16T08:03:42.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T08:03:42.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:03:42.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:03:42.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79228 2024-09-16T08:03:42.477 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T08:03:42.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T08:03:42.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T08:03:42.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-16T08:03:42.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-16T08:03:42.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-16T08:03:42.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T08:03:42.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T08:03:42.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T08:03:42.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-16T08:03:42.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-16T08:03:42.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T08:03:42.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:03:42.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:03:42.904 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:03:42.904 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:03:42.904 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:03:42.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:03:42.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T08:03:43.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T08:03:43.134 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T08:03:43.134 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T08:03:43.134 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T08:03:43.134 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T08:03:43.134 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T08:03:43.135 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T08:03:43.135 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T08:03:43.135 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T08:03:43.135 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:03:43.135 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:03:43.135 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:03:43.136 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T08:03:43.136 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T08:03:43.136 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get fsid 2024-09-16T08:03:43.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T08:03:43.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T08:03:43.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T08:03:43.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T08:03:43.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T08:03:43.239 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T08:03:43.239 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T08:03:43.239 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T08:03:43.240 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:03:43.240 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:03:43.240 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:03:43.240 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T08:03:43.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T08:03:43.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get mon_host 2024-09-16T08:03:43.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-16T08:03:43.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-16T08:03:43.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T08:03:43.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T08:03:43.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T08:03:43.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-16T08:03:43.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T08:03:44.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T08:03:44.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:03:44.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:03:44.277 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:03:44.277 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:03:44.277 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:03:44.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:03:44.278 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T08:03:44.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T08:03:44.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-16T08:03:44.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T08:03:44.631 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-16T08:03:44.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T08:03:45.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-16T08:03:45.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T08:03:45.644 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T08:03:45.644 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:03:45.644 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:03:45.644 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:03:45.645 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T08:03:45.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-16T08:03:45.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-mon.a.asok log flush 2024-09-16T08:03:45.727 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T08:03:45.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-16T08:03:45.738 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T08:03:42.957+0000 7f6d1c0cfd40 0 load: jerasure load: lrc 2024-09-16T08:03:45.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_single_recovery_error td/test-erasure-eio 2024-09-16T08:03:45.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:390: TEST_ec_single_recovery_error: local dir=td/test-erasure-eio 2024-09-16T08:03:45.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:391: TEST_ec_single_recovery_error: local objname=myobject 2024-09-16T08:03:45.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:393: TEST_ec_single_recovery_error: setup_osds 7 2024-09-16T08:03:45.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-09-16T08:03:45.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-16T08:03:45.740 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-09-16T08:03:45.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-09-16T08:03:45.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:03:45.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-16T08:03:45.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:03:45.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:03:45.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T08:03:45.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:03:45.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-16T08:03:45.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:03:45.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:03:45.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:03:45.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:03:45.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-16T08:03:45.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-16T08:03:45.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:03:45.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:03:45.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:03:45.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:03:45.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:03:45.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:03:45.750 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:03:45.750 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:03:45.750 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:03:45.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:03:45.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:03:45.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:03:45.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:03:45.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:03:45.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:03:45.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:03:45.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:03:45.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:03:45.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:03:45.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:03:45.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:03:45.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:03:45.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-16T08:03:45.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:03:45.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=29675d8f-5de7-4ebf-b449-5c75348701f4 2024-09-16T08:03:45.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 29675d8f-5de7-4ebf-b449-5c75348701f4' 2024-09-16T08:03:45.752 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 29675d8f-5de7-4ebf-b449-5c75348701f4 2024-09-16T08:03:45.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:03:45.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBh5udmSw8xLhAAej/LGxBDbhg91J2K0aTirw== 2024-09-16T08:03:45.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBh5udmSw8xLhAAej/LGxBDbhg91J2K0aTirw=="}' 2024-09-16T08:03:45.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 29675d8f-5de7-4ebf-b449-5c75348701f4 -i td/test-erasure-eio/0/new.json 2024-09-16T08:03:46.170 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:03:46.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-16T08:03:46.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBh5udmSw8xLhAAej/LGxBDbhg91J2K0aTirw== --osd-uuid 29675d8f-5de7-4ebf-b449-5c75348701f4 2024-09-16T08:03:46.220 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:03:46.217+0000 7f3f2e9f4880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:03:46.224 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:03:46.221+0000 7f3f2e9f4880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:03:46.226 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:03:46.225+0000 7f3f2e9f4880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:03:46.227 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:03:46.225+0000 7f3f2e9f4880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T08:03:46.227 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:03:46.225+0000 7f3f2e9f4880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T08:03:46.227 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:03:46.225+0000 7f3f2e9f4880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T08:03:46.332 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:03:46.329+0000 7f3f2e9f4880 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-16T08:03:52.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-16T08:03:52.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:03:52.137 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T08:03:52.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T08:03:52.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:03:52.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T08:03:52.732 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T08:03:52.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:03:52.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:03:52.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:03:52.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:03:52.766 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:03:52.765+0000 7f5a065f6880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:03:52.782 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:03:52.781+0000 7f5a065f6880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:03:52.784 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:03:52.781+0000 7f5a065f6880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:03:53.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T08:03:53.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:03:53.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T08:03:53.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:03:53.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:03:53.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:03:53.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:03:53.229 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:03:53.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:03:53.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:03:53.340 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:03:53.337+0000 7f5a065f6880 -1 Falling back to public interface 2024-09-16T08:03:53.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:03:54.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:03:54.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:03:54.660 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:03:54.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:03:54.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:03:54.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:03:55.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:03:55.926 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:03:55.925+0000 7f5a065f6880 -1 osd.0 0 log_to_monitors true 2024-09-16T08:03:56.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:03:56.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:03:56.134 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:03:56.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:03:56.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:03:56.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:03:56.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:03:57.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:03:57.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:03:57.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:03:57.663 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:03:57.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:03:57.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:03:58.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:03:59.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:03:59.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:03:59.100 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:03:59.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:03:59.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:03:59.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:03:59.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:00.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:00.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:00.674 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:04:00.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:04:00.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:00.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:04:01.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:02.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:02.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:02.133 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:04:02.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:04:02.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:02.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:04:02.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:03.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:03.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:03.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:04:03.705 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:04:03.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:03.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:04:04.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:05.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:05.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:05.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:04:05.132 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:04:05.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:05.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:04:05.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:06.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:06.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:06.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:04:06.702 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:04:06.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:06.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:04:07.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:08.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:08.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:08.136 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:04:08.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:04:08.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:08.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:04:08.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:09.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:09.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:09.566 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:04:09.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:04:09.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:09.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:04:10.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:11.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:11.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:11.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:04:11.622 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:04:11.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:11.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:04:12.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:13.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:13.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:13.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:04:13.457 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:04:13.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:13.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:04:14.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:15.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:15.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:15.650 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:04:15.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:04:15.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:15.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:04:16.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:17.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:17.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:17.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:04:17.561 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:04:17.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:17.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:04:18.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:19.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:19.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:19.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:04:19.305 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:04:19.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:19.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:04:19.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:20.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:20.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:20.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:04:20.880 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:04:20.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:20.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:04:21.408 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3279619679,v1:127.0.0.1:6803/3279619679] [v2:127.0.0.1:6804/3279619679,v1:127.0.0.1:6805/3279619679] exists,up 29675d8f-5de7-4ebf-b449-5c75348701f4 2024-09-16T08:04:21.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:04:21.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:04:21.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:04:21.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:04:21.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-16T08:04:21.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:04:21.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:04:21.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T08:04:21.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:04:21.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-16T08:04:21.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:04:21.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:04:21.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:04:21.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:04:21.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-16T08:04:21.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-16T08:04:21.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:04:21.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:04:21.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:04:21.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:04:21.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:04:21.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:04:21.413 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:04:21.413 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:04:21.413 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:04:21.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:04:21.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:04:21.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:04:21.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:04:21.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:04:21.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:04:21.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:04:21.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:04:21.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:04:21.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:04:21.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:04:21.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:04:21.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:04:21.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-16T08:04:21.416 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:04:21.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1d968f12-38a7-418a-bdde-eac6c3bce0ac 2024-09-16T08:04:21.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1d968f12-38a7-418a-bdde-eac6c3bce0ac' 2024-09-16T08:04:21.416 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 1d968f12-38a7-418a-bdde-eac6c3bce0ac 2024-09-16T08:04:21.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:04:21.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCF5udm42lsGhAAbcYs4gqlmZx3I7DkRzKi6g== 2024-09-16T08:04:21.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCF5udm42lsGhAAbcYs4gqlmZx3I7DkRzKi6g=="}' 2024-09-16T08:04:21.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1d968f12-38a7-418a-bdde-eac6c3bce0ac -i td/test-erasure-eio/1/new.json 2024-09-16T08:04:22.042 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:04:22.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-16T08:04:22.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCF5udm42lsGhAAbcYs4gqlmZx3I7DkRzKi6g== --osd-uuid 1d968f12-38a7-418a-bdde-eac6c3bce0ac 2024-09-16T08:04:22.102 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:04:22.101+0000 7f9ce1306880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:04:22.119 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:04:22.117+0000 7f9ce1306880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:04:22.121 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:04:22.117+0000 7f9ce1306880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:04:22.122 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:04:22.121+0000 7f9ce1306880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T08:04:22.122 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:04:22.121+0000 7f9ce1306880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T08:04:22.122 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:04:22.121+0000 7f9ce1306880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T08:04:22.210 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:04:22.209+0000 7f9ce1306880 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-16T08:04:28.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-16T08:04:28.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:04:28.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T08:04:28.293 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-16T08:04:28.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:04:28.867 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-16T08:04:28.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T08:04:28.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:04:28.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:04:28.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:04:28.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:04:28.902 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:04:28.900+0000 7fd3a6ae7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:04:28.905 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:04:28.900+0000 7fd3a6ae7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:04:28.907 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:04:28.904+0000 7fd3a6ae7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:04:29.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T08:04:29.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:04:29.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T08:04:29.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:04:29.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:04:29.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:29.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:04:29.297 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:04:29.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:29.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:04:29.507 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:04:29.504+0000 7fd3a6ae7880 -1 Falling back to public interface 2024-09-16T08:04:29.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:30.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:30.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:30.766 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:04:30.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:04:30.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:30.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:04:31.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:32.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:32.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:32.193 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:04:32.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:04:32.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:32.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:04:32.243 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:04:32.240+0000 7fd3a6ae7880 -1 osd.1 0 log_to_monitors true 2024-09-16T08:04:32.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:33.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:33.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:33.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:04:33.739 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:04:33.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:33.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:04:34.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:35.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:35.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:35.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:04:35.288 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:04:35.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:35.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:04:35.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:36.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:36.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:36.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:04:36.751 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:04:36.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:36.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:04:37.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:38.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:38.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:38.195 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:04:38.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:04:38.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:38.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:04:38.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:39.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:39.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:39.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:04:39.770 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:04:39.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:39.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:04:40.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:41.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:41.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:41.277 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:04:41.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:04:41.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:41.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:04:41.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:42.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:42.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:42.714 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:04:42.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:04:42.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:42.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:04:43.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:44.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:44.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:44.195 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:04:44.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:04:44.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:44.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:04:44.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:45.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:45.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:45.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:04:45.703 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:04:45.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:45.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:04:46.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:47.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:47.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:47.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:04:47.599 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:04:47.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:47.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:04:48.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:49.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:49.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:49.397 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:04:49.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:04:49.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:49.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:04:49.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:50.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:50.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:50.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:04:50.981 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:04:50.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:50.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:04:52.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:53.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:53.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:53.109 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:04:53.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:04:53.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:53.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:04:54.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:55.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:55.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:55.057 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:04:55.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:04:55.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:55.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:04:55.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:56.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:56.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:56.671 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:04:56.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:04:56.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:56.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:04:57.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:04:58.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:04:58.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:04:58.103 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-16T08:04:58.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T08:04:58.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:04:58.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:04:58.623 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 14 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2268089760,v1:127.0.0.1:6811/2268089760] [v2:127.0.0.1:6812/2268089760,v1:127.0.0.1:6813/2268089760] exists,up 1d968f12-38a7-418a-bdde-eac6c3bce0ac 2024-09-16T08:04:58.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:04:58.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:04:58.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:04:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:04:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-16T08:04:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:04:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:04:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T08:04:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:04:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-16T08:04:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:04:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:04:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:04:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:04:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-16T08:04:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-16T08:04:58.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:04:58.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:04:58.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:04:58.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:04:58.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:04:58.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:04:58.628 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:04:58.628 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:04:58.628 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:04:58.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:04:58.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:04:58.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:04:58.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:04:58.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:04:58.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:04:58.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:04:58.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:04:58.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:04:58.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:04:58.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:04:58.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:04:58.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:04:58.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-16T08:04:58.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:04:58.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d9599619-6317-4652-a208-a6797bc5e029 2024-09-16T08:04:58.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d9599619-6317-4652-a208-a6797bc5e029' 2024-09-16T08:04:58.632 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 d9599619-6317-4652-a208-a6797bc5e029 2024-09-16T08:04:58.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:04:58.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCq5udmM4g4JxAA0nC6pPfFuOuGSXrzy/Qfvg== 2024-09-16T08:04:58.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCq5udmM4g4JxAA0nC6pPfFuOuGSXrzy/Qfvg=="}' 2024-09-16T08:04:58.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d9599619-6317-4652-a208-a6797bc5e029 -i td/test-erasure-eio/2/new.json 2024-09-16T08:04:59.303 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:04:59.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-16T08:04:59.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCq5udmM4g4JxAA0nC6pPfFuOuGSXrzy/Qfvg== --osd-uuid d9599619-6317-4652-a208-a6797bc5e029 2024-09-16T08:04:59.359 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:04:59.356+0000 7feb8df0c880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:04:59.393 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:04:59.388+0000 7feb8df0c880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:04:59.395 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:04:59.392+0000 7feb8df0c880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:04:59.396 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:04:59.392+0000 7feb8df0c880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:04:59.396 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:04:59.392+0000 7feb8df0c880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:04:59.396 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:04:59.392+0000 7feb8df0c880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:04:59.484 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:04:59.480+0000 7feb8df0c880 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-16T08:05:05.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-16T08:05:05.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:05:05.948 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-16T08:05:05.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T08:05:05.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:05:06.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T08:05:06.513 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-16T08:05:06.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:05:06.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:05:06.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:05:06.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:05:06.549 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:05:06.544+0000 7f5fdb12a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:05:06.585 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:05:06.580+0000 7f5fdb12a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:05:06.586 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:05:06.584+0000 7f5fdb12a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:05:06.888 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:05:06.884+0000 7f5fdb12a880 -1 Falling back to public interface 2024-09-16T08:05:06.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T08:05:06.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:05:06.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T08:05:06.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:05:06.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:05:06.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:06.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:05:06.926 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:05:06.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:06.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:05:07.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:08.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:08.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:08.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:05:08.391 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:05:08.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:08.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:05:08.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:09.581 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:05:09.576+0000 7f5fdb12a880 -1 osd.2 0 log_to_monitors true 2024-09-16T08:05:09.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:09.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:09.827 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:05:09.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:05:09.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:09.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:05:10.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:11.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:11.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:11.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:05:11.461 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:05:11.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:05:11.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:11.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:12.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:12.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:12.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:05:12.912 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:05:12.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:12.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:05:13.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:14.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:14.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:14.450 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:05:14.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:05:14.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:14.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:05:14.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:15.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:15.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:15.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:05:15.959 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:05:15.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:15.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:05:16.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:17.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:17.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:17.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:05:17.433 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:05:17.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:17.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:05:18.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:19.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:19.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:19.053 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:05:19.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:05:19.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:19.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:05:19.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:20.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:20.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:20.572 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:05:20.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:05:20.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:20.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:05:21.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:22.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:22.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:22.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:05:22.038 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:05:22.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:22.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:05:22.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:23.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:23.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:23.597 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:05:23.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:05:23.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:23.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:05:24.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:25.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:25.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:25.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:05:25.414 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:05:25.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:05:25.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:26.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:27.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:27.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:27.624 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:05:27.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:05:27.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:27.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:05:28.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:29.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:29.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:29.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:05:29.524 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:05:29.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:29.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:05:30.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:31.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:31.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:31.360 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:05:31.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:05:31.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:31.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:05:32.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:33.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:33.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:33.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:05:33.182 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:05:33.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:33.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:05:33.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:34.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:34.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:34.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:05:34.882 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:05:34.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:34.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:05:35.701 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 21 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3769301755,v1:127.0.0.1:6819/3769301755] [v2:127.0.0.1:6820/3769301755,v1:127.0.0.1:6821/3769301755] exists,up d9599619-6317-4652-a208-a6797bc5e029 2024-09-16T08:05:35.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:05:35.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:05:35.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:05:35.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:05:35.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-16T08:05:35.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:05:35.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:05:35.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T08:05:35.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:05:35.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-16T08:05:35.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:05:35.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:05:35.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:05:35.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:05:35.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-16T08:05:35.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-16T08:05:35.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:05:35.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:05:35.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:05:35.704 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:05:35.705 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:05:35.705 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:05:35.706 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:05:35.706 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:05:35.707 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:05:35.707 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:05:35.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:05:35.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:05:35.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:05:35.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:05:35.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:05:35.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:05:35.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:05:35.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:05:35.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:05:35.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:05:35.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:05:35.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:05:35.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-16T08:05:35.709 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:05:35.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=474ccaff-7d1d-47d2-8366-b34acc689239 2024-09-16T08:05:35.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 474ccaff-7d1d-47d2-8366-b34acc689239' 2024-09-16T08:05:35.711 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 474ccaff-7d1d-47d2-8366-b34acc689239 2024-09-16T08:05:35.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:05:35.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDP5udmu88SLBAA0uZuxy33eaVVbc92FDlarQ== 2024-09-16T08:05:35.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDP5udmu88SLBAA0uZuxy33eaVVbc92FDlarQ=="}' 2024-09-16T08:05:35.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 474ccaff-7d1d-47d2-8366-b34acc689239 -i td/test-erasure-eio/3/new.json 2024-09-16T08:05:36.540 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:05:36.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-16T08:05:36.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDP5udmu88SLBAA0uZuxy33eaVVbc92FDlarQ== --osd-uuid 474ccaff-7d1d-47d2-8366-b34acc689239 2024-09-16T08:05:36.595 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:05:36.591+0000 7facc6473880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:05:36.629 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:05:36.623+0000 7facc6473880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:05:36.631 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:05:36.627+0000 7facc6473880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:05:36.632 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:05:36.627+0000 7facc6473880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:05:36.632 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:05:36.627+0000 7facc6473880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:05:36.633 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:05:36.627+0000 7facc6473880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:05:36.720 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:05:36.715+0000 7facc6473880 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-16T08:05:43.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-16T08:05:43.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:05:43.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T08:05:43.257 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-16T08:05:43.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:05:43.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T08:05:43.850 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-16T08:05:43.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:05:43.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:05:43.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:05:43.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:05:43.883 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:05:43.879+0000 7f48d6ed7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:05:43.886 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:05:43.883+0000 7f48d6ed7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:05:43.890 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:05:43.883+0000 7f48d6ed7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:05:44.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T08:05:44.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:05:44.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T08:05:44.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:05:44.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:05:44.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:44.336 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:05:44.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:05:44.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:44.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:05:44.467 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:05:44.463+0000 7f48d6ed7880 -1 Falling back to public interface 2024-09-16T08:05:44.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:45.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:45.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:45.771 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:05:45.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:05:45.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:45.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:05:46.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:47.142 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:05:47.139+0000 7f48d6ed7880 -1 osd.3 0 log_to_monitors true 2024-09-16T08:05:47.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:47.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:47.252 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:05:47.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:05:47.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:47.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:05:47.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:48.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:48.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:48.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:05:48.996 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:05:48.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:48.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:05:49.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:50.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:50.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:50.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:05:50.659 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:05:50.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:50.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:05:51.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:52.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:52.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:52.110 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:05:52.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:05:52.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:52.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:05:52.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:53.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:53.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:53.655 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:05:53.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:05:53.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:53.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:05:54.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:55.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:55.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:55.202 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:05:55.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:05:55.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:55.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:05:55.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:56.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:56.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:56.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:05:56.635 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:05:56.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:56.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:05:57.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:58.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:58.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:58.191 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:05:58.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:05:58.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:58.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:05:58.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:05:59.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:05:59.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:05:59.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:05:59.733 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:05:59.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:05:59.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:06:00.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:01.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:01.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:01.232 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:06:01.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:06:01.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:01.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:06:01.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:02.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:02.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:02.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:06:02.741 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:06:02.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:02.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:06:03.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:04.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:04.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:04.247 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:06:04.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:06:04.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:04.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:06:04.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:05.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:05.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:05.929 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:06:05.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:06:05.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:05.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:06:06.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:07.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:07.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:07.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:06:07.763 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:06:07.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:07.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:06:08.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:09.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:09.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:09.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:06:09.542 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:06:09.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:09.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:06:10.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:11.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:11.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:11.458 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:06:11.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:06:11.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:11.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:06:12.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:13.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:13.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:13.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T08:06:13.318 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-16T08:06:13.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:13.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:06:13.749 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3073603232,v1:127.0.0.1:6827/3073603232] [v2:127.0.0.1:6828/3073603232,v1:127.0.0.1:6829/3073603232] exists,up 474ccaff-7d1d-47d2-8366-b34acc689239 2024-09-16T08:06:13.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:06:13.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:06:13.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:06:13.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:06:13.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-16T08:06:13.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:06:13.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:06:13.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-16T08:06:13.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:06:13.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-16T08:06:13.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:06:13.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:06:13.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:06:13.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:06:13.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-16T08:06:13.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-16T08:06:13.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:06:13.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:06:13.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:06:13.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:06:13.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:06:13.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:06:13.757 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:06:13.757 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:06:13.757 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:06:13.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:06:13.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:06:13.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:06:13.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:06:13.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:06:13.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:06:13.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:06:13.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:06:13.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:06:13.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:06:13.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:06:13.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:06:13.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:06:13.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-16T08:06:13.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:06:13.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=65e1e34c-6057-4432-9df8-0623e46d85a8 2024-09-16T08:06:13.759 INFO:tasks.workunit.client.0.smithi162.stdout:add osd4 65e1e34c-6057-4432-9df8-0623e46d85a8 2024-09-16T08:06:13.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 65e1e34c-6057-4432-9df8-0623e46d85a8' 2024-09-16T08:06:13.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:06:13.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD15udm2Yu+LhAAlUlcWCiEL5iDqG5oU1dfTQ== 2024-09-16T08:06:13.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD15udm2Yu+LhAAlUlcWCiEL5iDqG5oU1dfTQ=="}' 2024-09-16T08:06:13.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 65e1e34c-6057-4432-9df8-0623e46d85a8 -i td/test-erasure-eio/4/new.json 2024-09-16T08:06:14.968 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:06:14.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-16T08:06:14.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD15udm2Yu+LhAAlUlcWCiEL5iDqG5oU1dfTQ== --osd-uuid 65e1e34c-6057-4432-9df8-0623e46d85a8 2024-09-16T08:06:15.028 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:06:15.022+0000 7fdc87343880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:06:15.032 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:06:15.026+0000 7fdc87343880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:06:15.034 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:06:15.030+0000 7fdc87343880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:06:15.035 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:06:15.030+0000 7fdc87343880 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-16T08:06:15.035 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:06:15.030+0000 7fdc87343880 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-16T08:06:15.035 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:06:15.030+0000 7fdc87343880 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-16T08:06:15.219 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:06:15.218+0000 7fdc87343880 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-16T08:06:21.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-16T08:06:21.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:06:21.712 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd4 key to auth repository 2024-09-16T08:06:21.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-16T08:06:21.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:06:22.278 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.4 2024-09-16T08:06:22.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-16T08:06:22.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:06:22.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:06:22.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:06:22.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:06:22.312 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:06:22.310+0000 7f806b54d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:06:22.322 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:06:22.322+0000 7f806b54d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:06:22.323 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:06:22.322+0000 7f806b54d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:06:22.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-16T08:06:22.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:06:22.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-16T08:06:22.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:06:22.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:06:22.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:22.817 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:06:22.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:06:22.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:22.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:06:22.879 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:06:22.878+0000 7f806b54d880 -1 Falling back to public interface 2024-09-16T08:06:23.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:24.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:24.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:24.322 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:06:24.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:06:24.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:24.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:06:24.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:25.522 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:06:25.522+0000 7f806b54d880 -1 osd.4 0 log_to_monitors true 2024-09-16T08:06:25.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:25.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:25.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:06:25.798 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:06:25.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:25.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:06:26.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:27.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:27.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:27.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:06:27.610 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:06:27.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:27.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:06:28.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:29.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:29.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:29.257 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:06:29.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:06:29.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:29.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:06:29.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:30.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:30.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:30.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:06:30.772 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:06:30.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:30.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:06:31.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:32.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:32.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:32.341 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:06:32.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:06:32.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:32.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:06:32.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:33.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:33.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:33.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:06:33.809 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:06:33.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:33.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:06:34.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:35.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:35.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:35.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:06:35.411 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:06:35.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:35.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:06:35.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:36.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:36.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:36.833 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:06:36.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:06:36.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:36.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:06:37.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:38.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:38.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:38.341 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:06:38.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:06:38.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:38.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:06:38.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:39.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:39.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:39.834 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:06:39.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:06:39.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:39.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:06:40.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:41.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:41.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:41.955 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:06:41.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:06:41.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:41.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:06:42.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:43.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:43.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:43.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:06:43.957 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:06:43.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:43.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:06:44.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:45.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:45.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:45.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:06:45.846 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:06:45.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:45.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:06:46.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:47.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:47.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:47.546 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:06:47.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:06:47.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:47.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:06:48.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:49.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:49.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:49.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:06:49.526 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:06:49.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:49.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:06:50.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:06:51.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:06:51.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:06:51.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:06:51.625 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:06:51.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:06:51.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-16T08:06:52.085 INFO:tasks.workunit.client.0.smithi162.stdout:osd.4 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2420281006,v1:127.0.0.1:6835/2420281006] [v2:127.0.0.1:6836/2420281006,v1:127.0.0.1:6837/2420281006] exists,up 65e1e34c-6057-4432-9df8-0623e46d85a8 2024-09-16T08:06:52.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:06:52.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:06:52.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:06:52.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:06:52.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-16T08:06:52.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:06:52.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:06:52.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-16T08:06:52.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:06:52.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-16T08:06:52.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:06:52.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:06:52.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:06:52.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:06:52.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-16T08:06:52.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-16T08:06:52.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:06:52.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:06:52.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:06:52.089 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:06:52.089 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:06:52.089 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:06:52.090 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:06:52.090 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:06:52.090 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:06:52.091 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:06:52.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:06:52.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:06:52.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:06:52.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:06:52.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:06:52.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:06:52.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:06:52.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:06:52.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:06:52.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:06:52.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:06:52.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:06:52.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-16T08:06:52.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:06:52.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d7d717d4-5803-4256-952e-07cd0fce135b 2024-09-16T08:06:52.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 d7d717d4-5803-4256-952e-07cd0fce135b' 2024-09-16T08:06:52.093 INFO:tasks.workunit.client.0.smithi162.stdout:add osd5 d7d717d4-5803-4256-952e-07cd0fce135b 2024-09-16T08:06:52.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:06:52.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAc5+dmqBUoBxAAeagHlfQsdeN3WNC4CkECpA== 2024-09-16T08:06:52.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAc5+dmqBUoBxAAeagHlfQsdeN3WNC4CkECpA=="}' 2024-09-16T08:06:52.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d7d717d4-5803-4256-952e-07cd0fce135b -i td/test-erasure-eio/5/new.json 2024-09-16T08:06:52.812 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:06:52.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-16T08:06:52.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAc5+dmqBUoBxAAeagHlfQsdeN3WNC4CkECpA== --osd-uuid d7d717d4-5803-4256-952e-07cd0fce135b 2024-09-16T08:06:52.869 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:06:52.866+0000 7f0e6bafb880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:06:52.925 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:06:52.922+0000 7f0e6bafb880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:06:52.925 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:06:52.922+0000 7f0e6bafb880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:06:52.926 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:06:52.926+0000 7f0e6bafb880 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-16T08:06:52.926 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:06:52.926+0000 7f0e6bafb880 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-16T08:06:52.926 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:06:52.926+0000 7f0e6bafb880 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-16T08:06:53.078 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:06:53.078+0000 7f0e6bafb880 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-16T08:07:01.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-16T08:07:01.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:07:01.865 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd5 key to auth repository 2024-09-16T08:07:01.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-16T08:07:01.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:07:02.436 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.5 2024-09-16T08:07:02.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-16T08:07:02.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:07:02.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:07:02.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:07:02.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:07:02.463 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:07:02.461+0000 7f76cb7ac880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:07:02.485 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:07:02.481+0000 7f76cb7ac880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:07:02.487 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:07:02.485+0000 7f76cb7ac880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:07:02.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-16T08:07:02.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:07:02.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-16T08:07:02.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:07:02.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:07:02.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:02.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:07:02.918 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:07:02.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T08:07:02.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:03.071 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:07:03.069+0000 7f76cb7ac880 -1 Falling back to public interface 2024-09-16T08:07:03.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:04.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:04.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:04.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:07:04.379 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:07:04.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:04.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T08:07:04.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:05.571 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:07:05.569+0000 7f76cb7ac880 -1 osd.5 0 log_to_monitors true 2024-09-16T08:07:05.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:05.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:05.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:07:05.839 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:07:05.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:05.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T08:07:06.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:07.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:07.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:07.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:07:07.594 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:07:07.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:07.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T08:07:08.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:09.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:09.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:09.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:07:09.297 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:07:09.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:09.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T08:07:09.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:10.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:10.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:10.850 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:07:10.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:07:10.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:10.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T08:07:11.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:12.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:12.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:12.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:07:12.369 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:07:12.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:12.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T08:07:12.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:13.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:13.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:13.874 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:07:13.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:07:13.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:13.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T08:07:14.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:15.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:15.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:15.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:07:15.452 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:07:15.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:15.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T08:07:15.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:16.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:16.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:16.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:07:16.905 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:07:16.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:16.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T08:07:17.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:18.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:18.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:18.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:07:18.367 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:07:18.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:18.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T08:07:18.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:19.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:19.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:19.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:07:19.819 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:07:19.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:19.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T08:07:20.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:21.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:21.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:21.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:07:21.769 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:07:21.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:21.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T08:07:22.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:23.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:23.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:23.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:07:23.622 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:07:23.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:23.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T08:07:24.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:25.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:25.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:25.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:07:25.411 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:07:25.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:25.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T08:07:26.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:27.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:27.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:27.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:07:27.255 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:07:27.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:27.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T08:07:28.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:29.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:29.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:29.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:07:29.130 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:07:29.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:29.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T08:07:30.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:31.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:31.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:31.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:07:31.059 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:07:31.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:31.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T08:07:31.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:32.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:32.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:32.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T08:07:32.544 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-16T08:07:32.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:32.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-16T08:07:33.038 INFO:tasks.workunit.client.0.smithi162.stdout:osd.5 up in weight 1 up_from 52 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/185661701,v1:127.0.0.1:6843/185661701] [v2:127.0.0.1:6844/185661701,v1:127.0.0.1:6845/185661701] exists,up d7d717d4-5803-4256-952e-07cd0fce135b 2024-09-16T08:07:33.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:07:33.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:07:33.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:07:33.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:07:33.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-16T08:07:33.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:07:33.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:07:33.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-09-16T08:07:33.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:07:33.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-16T08:07:33.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:07:33.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:07:33.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:07:33.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:07:33.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-16T08:07:33.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-16T08:07:33.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:07:33.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:07:33.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:07:33.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:07:33.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:07:33.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:07:33.044 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:07:33.044 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:07:33.044 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:07:33.044 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:07:33.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:07:33.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:07:33.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:07:33.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:07:33.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:07:33.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:07:33.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:07:33.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:07:33.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:07:33.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:07:33.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:07:33.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:07:33.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-16T08:07:33.046 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:07:33.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=405f475e-bfe8-4012-aad6-6ae0009fa254 2024-09-16T08:07:33.047 INFO:tasks.workunit.client.0.smithi162.stdout:add osd6 405f475e-bfe8-4012-aad6-6ae0009fa254 2024-09-16T08:07:33.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 405f475e-bfe8-4012-aad6-6ae0009fa254' 2024-09-16T08:07:33.048 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:07:33.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBF5+dmS4FeBBAALUQMl14c4+0GF0nSdQz+3g== 2024-09-16T08:07:33.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBF5+dmS4FeBBAALUQMl14c4+0GF0nSdQz+3g=="}' 2024-09-16T08:07:33.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 405f475e-bfe8-4012-aad6-6ae0009fa254 -i td/test-erasure-eio/6/new.json 2024-09-16T08:07:34.155 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:07:34.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-16T08:07:34.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBF5+dmS4FeBBAALUQMl14c4+0GF0nSdQz+3g== --osd-uuid 405f475e-bfe8-4012-aad6-6ae0009fa254 2024-09-16T08:07:34.212 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:07:34.209+0000 7f0ef7af2880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:07:34.216 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:07:34.213+0000 7f0ef7af2880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:07:34.218 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:07:34.217+0000 7f0ef7af2880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:07:34.219 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:07:34.217+0000 7f0ef7af2880 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-16T08:07:34.219 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:07:34.217+0000 7f0ef7af2880 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-16T08:07:34.219 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:07:34.217+0000 7f0ef7af2880 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-16T08:07:34.336 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:07:34.337+0000 7f0ef7af2880 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-16T08:07:41.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-16T08:07:41.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:07:41.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-09-16T08:07:41.298 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd6 key to auth repository 2024-09-16T08:07:41.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:07:41.866 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.6 2024-09-16T08:07:41.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-09-16T08:07:41.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:07:41.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:07:41.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:07:41.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:07:41.898 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:07:41.897+0000 7f9ee1504880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:07:41.920 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:07:41.917+0000 7f9ee1504880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:07:41.922 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:07:41.921+0000 7f9ee1504880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:07:42.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-09-16T08:07:42.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:07:42.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-09-16T08:07:42.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:07:42.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:07:42.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:42.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:07:42.294 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:07:42.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:42.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T08:07:42.480 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:07:42.477+0000 7f9ee1504880 -1 Falling back to public interface 2024-09-16T08:07:42.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:43.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:43.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:43.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:07:43.733 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:07:43.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:43.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T08:07:44.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:45.122 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:07:45.121+0000 7f9ee1504880 -1 osd.6 0 log_to_monitors true 2024-09-16T08:07:45.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:45.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:45.193 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:07:45.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:07:45.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:45.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T08:07:45.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:46.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:46.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:46.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:07:46.930 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:07:46.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:46.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T08:07:47.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:48.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:48.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:48.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:07:48.403 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:07:48.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:48.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T08:07:49.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:50.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:50.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:50.058 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:07:50.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:07:50.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:50.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T08:07:50.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:51.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:51.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:51.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:07:51.487 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:07:51.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:51.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T08:07:52.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:53.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:53.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:53.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:07:53.166 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:07:53.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:53.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T08:07:53.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:54.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:54.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:54.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:07:54.628 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:07:54.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:54.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T08:07:55.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:56.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:56.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:56.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:07:56.166 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:07:56.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:56.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T08:07:56.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:57.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:57.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:57.723 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:07:57.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:07:57.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:57.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T08:07:58.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:07:59.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:07:59.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:07:59.287 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:07:59.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:07:59.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:07:59.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T08:07:59.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:08:00.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:08:00.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:08:00.711 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:08:00.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:08:00.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:08:00.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T08:08:01.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:08:02.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:08:02.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:08:02.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:08:02.399 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:08:02.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:08:02.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T08:08:03.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:08:04.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:08:04.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:08:04.058 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:08:04.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:08:04.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:08:04.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T08:08:05.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:08:06.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:08:06.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:08:06.103 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:08:06.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:08:06.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:08:06.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T08:08:06.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:08:07.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:08:07.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:08:07.940 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:08:07.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:08:07.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:08:07.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T08:08:08.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:08:09.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:08:09.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:08:09.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:08:09.840 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:08:09.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:08:09.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T08:08:10.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:08:11.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:08:11.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:08:11.803 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-16T08:08:11.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T08:08:11.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:08:11.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-16T08:08:12.462 INFO:tasks.workunit.client.0.smithi162.stdout:osd.6 up in weight 1 up_from 62 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3952716010,v1:127.0.0.1:6851/3952716010] [v2:127.0.0.1:6852/3952716010,v1:127.0.0.1:6853/3952716010] exists,up 405f475e-bfe8-4012-aad6-6ae0009fa254 2024-09-16T08:08:12.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:08:12.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:08:12.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:08:12.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-16T08:08:12.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T08:08:12.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T08:08:12.465 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:08:12.465 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:08:12.465 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:08:12.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.0.asok 2024-09-16T08:08:12.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-16T08:08:12.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.0.asok log flush 2024-09-16T08:08:12.565 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T08:08:12.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-16T08:08:12.578 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T08:03:53.349+0000 7f5a065f6880 0 load: jerasure load: lrc 2024-09-16T08:08:12.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:395: TEST_ec_single_recovery_error: local poolname=pool-jerasure 2024-09-16T08:08:12.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:396: TEST_ec_single_recovery_error: create_erasure_coded_pool pool-jerasure 3 2 2024-09-16T08:08:12.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T08:08:12.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-16T08:08:12.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-16T08:08:12.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-16T08:08:12.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-16T08:08:12.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-16T08:08:12.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-16T08:08:13.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-16T08:08:13.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-16T08:08:15.242 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-16T08:08:15.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T08:08:16.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-16T08:08:16.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:08:16.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:08:16.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:08:16.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:08:16.257 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:08:16.258 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:08:16.258 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:08:16.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:08:16.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:08:16.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:08:16.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:08:16.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:08:16.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:08:16.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:08:16.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:08:16.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:08:16.913 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:08:16.913 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:08:16.913 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T08:08:16.913 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T08:08:16.913 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T08:08:16.913 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-16T08:08:16.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:08:16.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:08:16.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:08:17.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803832 2024-09-16T08:08:17.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803832 2024-09-16T08:08:17.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803832' 2024-09-16T08:08:17.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:08:17.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:08:17.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=60129542195 2024-09-16T08:08:17.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 60129542195 2024-09-16T08:08:17.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803832 1-60129542195' 2024-09-16T08:08:17.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:08:17.506 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:08:17.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=90194313260 2024-09-16T08:08:17.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 90194313260 2024-09-16T08:08:17.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803832 1-60129542195 2-90194313260' 2024-09-16T08:08:17.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:08:17.672 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:08:17.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=133143986216 2024-09-16T08:08:17.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 133143986216 2024-09-16T08:08:17.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803832 1-60129542195 2-90194313260 3-133143986216' 2024-09-16T08:08:17.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:08:17.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T08:08:17.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=171798691873 2024-09-16T08:08:17.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 171798691873 2024-09-16T08:08:17.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803832 1-60129542195 2-90194313260 3-133143986216 4-171798691873' 2024-09-16T08:08:17.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:08:17.990 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T08:08:18.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=223338299414 2024-09-16T08:08:18.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 223338299414 2024-09-16T08:08:18.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803832 1-60129542195 2-90194313260 3-133143986216 4-171798691873 5-223338299414' 2024-09-16T08:08:18.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:08:18.145 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-16T08:08:18.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=266287972363 2024-09-16T08:08:18.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 266287972363 2024-09-16T08:08:18.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803832 1-60129542195 2-90194313260 3-133143986216 4-171798691873 5-223338299414 6-266287972363' 2024-09-16T08:08:18.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:08:18.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803832 2024-09-16T08:08:18.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:08:18.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:08:18.313 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803832 2024-09-16T08:08:18.314 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:08:18.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803832 2024-09-16T08:08:18.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803832' 2024-09-16T08:08:18.316 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803832 2024-09-16T08:08:18.316 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:08:18.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803832 -lt 25769803832 2024-09-16T08:08:18.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:08:18.848 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-60129542195 2024-09-16T08:08:18.848 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:08:18.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:08:18.852 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-60129542195 2024-09-16T08:08:18.852 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:08:18.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=60129542195 2024-09-16T08:08:18.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 60129542195' 2024-09-16T08:08:18.854 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 60129542195 2024-09-16T08:08:18.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:08:19.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 60129542196 -lt 60129542195 2024-09-16T08:08:19.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:08:19.335 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-90194313260 2024-09-16T08:08:19.335 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:08:19.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:08:19.337 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-90194313260 2024-09-16T08:08:19.337 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:08:19.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=90194313260 2024-09-16T08:08:19.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 90194313260' 2024-09-16T08:08:19.339 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 90194313260 2024-09-16T08:08:19.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:08:19.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 90194313260 -lt 90194313260 2024-09-16T08:08:19.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:08:19.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:08:19.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-133143986216 2024-09-16T08:08:19.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:08:19.792 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-133143986216 2024-09-16T08:08:19.792 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:08:19.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=133143986216 2024-09-16T08:08:19.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 133143986216' 2024-09-16T08:08:19.795 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 133143986216 2024-09-16T08:08:19.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:08:20.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 133143986217 -lt 133143986216 2024-09-16T08:08:20.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:08:20.234 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-171798691873 2024-09-16T08:08:20.234 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:08:20.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T08:08:20.237 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:08:20.237 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-171798691873 2024-09-16T08:08:20.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=171798691873 2024-09-16T08:08:20.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 171798691873' 2024-09-16T08:08:20.239 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 171798691873 2024-09-16T08:08:20.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T08:08:20.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 171798691873 -lt 171798691873 2024-09-16T08:08:20.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:08:20.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-223338299414 2024-09-16T08:08:20.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:08:20.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T08:08:20.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-223338299414 2024-09-16T08:08:20.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:08:20.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=223338299414 2024-09-16T08:08:20.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 223338299414' 2024-09-16T08:08:20.755 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 223338299414 2024-09-16T08:08:20.755 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T08:08:21.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 223338299414 -lt 223338299414 2024-09-16T08:08:21.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:08:21.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 6-266287972363 2024-09-16T08:08:21.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:08:21.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=6 2024-09-16T08:08:21.220 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 6-266287972363 2024-09-16T08:08:21.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:08:21.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=266287972363 2024-09-16T08:08:21.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.6 seq 266287972363' 2024-09-16T08:08:21.223 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 266287972363 2024-09-16T08:08:21.224 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-16T08:08:21.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 266287972363 -lt 266287972363 2024-09-16T08:08:21.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:08:21.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:08:21.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:08:22.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:08:22.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:08:22.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:08:22.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:08:22.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:08:22.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:08:22.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:08:22.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:08:22.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:08:22.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:08:22.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:08:22.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:08:23.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:08:23.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:08:23.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:08:23.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:398: TEST_ec_single_recovery_error: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-16T08:08:23.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-16T08:08:23.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-16T08:08:23.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-16T08:08:23.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:08:23.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-16T08:08:23.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:08:23.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-16T08:08:23.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:08:23.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-16T08:08:23.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:08:23.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-16T08:08:23.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-16T08:08:23.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:399: TEST_ec_single_recovery_error: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-09-16T08:08:23.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=ec 2024-09-16T08:08:23.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T08:08:23.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T08:08:23.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T08:08:23.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-jerasure 2024-09-16T08:08:23.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T08:08:23.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=myobject 2024-09-16T08:08:23.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T08:08:23.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/test-erasure-eio 2024-09-16T08:08:23.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T08:08:23.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T08:08:23.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T08:08:23.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:08:23.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-jerasure myobject 2024-09-16T08:08:23.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:08:23.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-16T08:08:23.388 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-16T08:08:23.388 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:08:23.837 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:08:23.837 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T08:08:23.837 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:08:23.837 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T08:08:23.837 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-16T08:08:23.837 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2024-09-16T08:08:23.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T08:08:23.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T08:08:23.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' ec '!=' ec ']' 2024-09-16T08:08:23.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/test-erasure-eio/3/type 2024-09-16T08:08:23.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T08:08:23.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T08:08:23.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T08:08:23.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T08:08:23.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T08:08:23.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T08:08:23.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T08:08:23.841 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T08:08:23.841 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T08:08:23.841 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T08:08:23.841 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:08:23.841 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:08:23.841 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:08:23.842 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.3.asok 2024-09-16T08:08:23.842 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79228/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T08:08:23.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T08:08:23.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T08:08:23.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T08:08:23.946 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T08:08:23.946 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T08:08:23.946 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T08:08:23.946 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:08:23.947 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:08:23.947 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:08:23.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.3.asok 2024-09-16T08:08:23.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T08:08:23.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-09-16T08:08:24.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:08:24.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: get_osds pool-jerasure myobject 2024-09-16T08:08:24.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:08:24.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-16T08:08:24.048 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-16T08:08:24.048 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:08:24.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:08:24.471 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T08:08:24.471 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:08:24.471 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T08:08:24.472 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-16T08:08:24.472 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2024-09-16T08:08:24.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: local -a initial_osds 2024-09-16T08:08:24.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:402: TEST_ec_single_recovery_error: local last_osd=6 2024-09-16T08:08:24.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:404: TEST_ec_single_recovery_error: kill_daemons td/test-erasure-eio TERM osd.6 2024-09-16T08:08:24.472 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:08:24.472 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:08:24.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:08:24.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:08:24.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:08:24.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:08:24.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:405: TEST_ec_single_recovery_error: ceph osd down 6 2024-09-16T08:08:25.893 INFO:tasks.workunit.client.0.smithi162.stderr:osd.6 is already down. 2024-09-16T08:08:25.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:406: TEST_ec_single_recovery_error: ceph osd out 6 2024-09-16T08:08:26.679 INFO:tasks.workunit.client.0.smithi162.stderr:osd.6 is already out. 2024-09-16T08:08:26.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:409: TEST_ec_single_recovery_error: wait_for_clean 2024-09-16T08:08:26.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:08:26.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:08:26.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:08:26.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:08:26.689 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:08:26.690 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:08:26.690 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:08:26.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:08:26.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:08:26.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:08:26.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:08:26.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:08:26.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:08:26.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:08:26.901 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:08:27.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:08:27.509 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:08:27.509 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:08:27.509 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T08:08:27.509 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-16T08:08:27.510 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-16T08:08:27.510 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-16T08:08:27.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:08:27.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:08:27.510 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:08:27.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803836 2024-09-16T08:08:27.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803836 2024-09-16T08:08:27.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803836' 2024-09-16T08:08:27.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:08:27.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:08:27.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=60129542200 2024-09-16T08:08:27.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 60129542200 2024-09-16T08:08:27.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803836 1-60129542200' 2024-09-16T08:08:27.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:08:27.967 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:08:28.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=90194313264 2024-09-16T08:08:28.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 90194313264 2024-09-16T08:08:28.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803836 1-60129542200 2-90194313264' 2024-09-16T08:08:28.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:08:28.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:08:28.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=133143986220 2024-09-16T08:08:28.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 133143986220 2024-09-16T08:08:28.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803836 1-60129542200 2-90194313264 3-133143986220' 2024-09-16T08:08:28.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:08:28.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-16T08:08:28.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=171798691877 2024-09-16T08:08:28.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 171798691877 2024-09-16T08:08:28.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803836 1-60129542200 2-90194313264 3-133143986220 4-171798691877' 2024-09-16T08:08:28.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:08:28.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-16T08:08:28.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=223338299418 2024-09-16T08:08:28.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 223338299418 2024-09-16T08:08:28.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803836 1-60129542200 2-90194313264 3-133143986220 4-171798691877 5-223338299418' 2024-09-16T08:08:28.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:08:28.693 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-16T08:08:28.805 INFO:tasks.workunit.client.0.smithi162.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-16T08:08:28.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq= 2024-09-16T08:08:28.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z '' 2024-09-16T08:08:28.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: flush_pg_stats: continue 2024-09-16T08:08:28.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:08:28.809 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803836 2024-09-16T08:08:28.810 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:08:28.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:08:28.813 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803836 2024-09-16T08:08:28.813 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:08:28.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803836 2024-09-16T08:08:28.815 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803836 2024-09-16T08:08:28.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803836' 2024-09-16T08:08:28.815 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:08:29.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803836 -lt 25769803836 2024-09-16T08:08:29.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:08:29.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-60129542200 2024-09-16T08:08:29.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:08:29.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:08:29.339 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-60129542200 2024-09-16T08:08:29.339 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:08:29.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=60129542200 2024-09-16T08:08:29.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 60129542200' 2024-09-16T08:08:29.341 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 60129542200 2024-09-16T08:08:29.342 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:08:29.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 60129542198 -lt 60129542200 2024-09-16T08:08:29.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T08:08:30.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T08:08:30.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:08:31.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 60129542200 -lt 60129542200 2024-09-16T08:08:31.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:08:31.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-90194313264 2024-09-16T08:08:31.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:08:31.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:08:31.196 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-90194313264 2024-09-16T08:08:31.196 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:08:31.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=90194313264 2024-09-16T08:08:31.198 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 90194313264 2024-09-16T08:08:31.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 90194313264' 2024-09-16T08:08:31.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:08:31.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 90194313264 -lt 90194313264 2024-09-16T08:08:31.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:08:31.665 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-133143986220 2024-09-16T08:08:31.666 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:08:31.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:08:31.670 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-133143986220 2024-09-16T08:08:31.670 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:08:31.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=133143986220 2024-09-16T08:08:31.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 133143986220' 2024-09-16T08:08:31.671 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 133143986220 2024-09-16T08:08:31.672 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:08:32.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 133143986221 -lt 133143986220 2024-09-16T08:08:32.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:08:32.138 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-171798691877 2024-09-16T08:08:32.138 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:08:32.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2024-09-16T08:08:32.140 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-171798691877 2024-09-16T08:08:32.141 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:08:32.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=171798691877 2024-09-16T08:08:32.143 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 171798691877 2024-09-16T08:08:32.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 171798691877' 2024-09-16T08:08:32.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-16T08:08:32.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 171798691877 -lt 171798691877 2024-09-16T08:08:32.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:08:32.606 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-223338299418 2024-09-16T08:08:32.606 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:08:32.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2024-09-16T08:08:32.609 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-223338299418 2024-09-16T08:08:32.609 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:08:32.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=223338299418 2024-09-16T08:08:32.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 223338299418' 2024-09-16T08:08:32.612 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 223338299418 2024-09-16T08:08:32.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-16T08:08:33.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 223338299418 -lt 223338299418 2024-09-16T08:08:33.017 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:08:33.018 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:08:33.018 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:08:33.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:08:33.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:08:33.591 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:08:33.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:08:33.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:08:33.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:08:33.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:08:33.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:08:34.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:08:34.005 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:08:34.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:08:34.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:08:34.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:08:34.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:08:34.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:08:34.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:411: TEST_ec_single_recovery_error: rados_get td/test-erasure-eio pool-jerasure myobject 2024-09-16T08:08:34.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T08:08:34.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T08:08:34.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-09-16T08:08:34.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-16T08:08:34.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-16T08:08:34.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-09-16T08:08:34.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-16T08:08:34.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-16T08:08:34.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:413: TEST_ec_single_recovery_error: delete_erasure_coded_pool pool-jerasure 2024-09-16T08:08:34.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T08:08:34.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-16T08:08:35.232 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-16T08:08:35.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-16T08:08:36.051 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-16T08:08:36.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-16T08:08:36.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T08:08:36.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T08:08:36.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T08:08:36.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:08:36.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:08:36.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:08:36.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:08:36.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:08:36.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:08:36.389 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T08:08:36.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T08:08:36.392 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T08:08:36.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T08:08:36.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T08:08:36.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T08:08:36.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:08:36.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T08:08:36.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T08:08:36.398 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:08:36.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:08:36.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T08:08:36.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T08:08:36.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T08:08:36.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:08:36.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:08:36.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T08:08:36.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T08:08:36.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T08:08:36.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-16T08:08:36.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-16T08:08:36.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-16T08:08:36.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-16T08:08:36.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T08:08:36.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T08:08:36.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T08:08:36.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:08:36.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:08:36.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:08:36.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:08:36.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:08:36.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:08:36.447 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T08:08:36.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T08:08:36.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T08:08:36.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T08:08:36.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T08:08:36.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T08:08:36.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:08:36.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T08:08:36.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T08:08:36.455 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:08:36.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:08:36.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T08:08:36.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T08:08:36.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T08:08:36.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:08:36.462 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:08:36.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T08:08:36.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T08:08:36.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T08:08:36.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-16T08:08:36.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T08:08:36.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:08:36.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:08:36.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79228 2024-09-16T08:08:36.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T08:08:36.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T08:08:36.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T08:08:36.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-16T08:08:36.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-16T08:08:36.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-16T08:08:36.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T08:08:36.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T08:08:36.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T08:08:36.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-16T08:08:36.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-16T08:08:36.944 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T08:08:36.944 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:08:36.944 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:08:36.944 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:08:36.944 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:08:36.944 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:08:36.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:08:36.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T08:08:37.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T08:08:37.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T08:08:37.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T08:08:37.223 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T08:08:37.223 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T08:08:37.223 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T08:08:37.224 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T08:08:37.224 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T08:08:37.224 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T08:08:37.225 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:08:37.225 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:08:37.225 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:08:37.226 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T08:08:37.226 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T08:08:37.226 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get fsid 2024-09-16T08:08:37.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T08:08:37.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T08:08:37.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T08:08:37.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T08:08:37.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T08:08:37.330 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T08:08:37.330 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T08:08:37.330 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T08:08:37.330 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:08:37.330 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:08:37.330 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:08:37.331 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T08:08:37.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T08:08:37.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get mon_host 2024-09-16T08:08:37.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-16T08:08:37.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-16T08:08:37.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T08:08:37.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T08:08:37.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T08:08:37.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-16T08:08:37.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T08:08:38.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T08:08:38.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:08:38.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:08:38.329 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:08:38.329 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:08:38.329 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:08:38.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:08:38.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T08:08:38.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T08:08:38.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-16T08:08:38.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T08:08:38.719 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-16T08:08:38.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T08:08:39.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-16T08:08:39.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T08:08:39.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T08:08:39.732 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:08:39.732 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:08:39.732 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:08:39.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T08:08:39.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-16T08:08:39.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-mon.a.asok log flush 2024-09-16T08:08:39.816 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T08:08:39.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-16T08:08:39.827 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T08:08:36.995+0000 7fd51a84cd40 0 load: jerasure load: lrc 2024-09-16T08:08:39.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_0 td/test-erasure-eio 2024-09-16T08:08:39.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:302: TEST_rados_get_bad_size_shard_0: local dir=td/test-erasure-eio 2024-09-16T08:08:39.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:303: TEST_rados_get_bad_size_shard_0: setup_osds 4 2024-09-16T08:08:39.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-16T08:08:39.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-16T08:08:39.829 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-16T08:08:39.831 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-16T08:08:39.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:08:39.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-16T08:08:39.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:08:39.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:08:39.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T08:08:39.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:08:39.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-16T08:08:39.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:08:39.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:08:39.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:08:39.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:08:39.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-16T08:08:39.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-16T08:08:39.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:08:39.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:08:39.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:08:39.834 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:08:39.834 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:08:39.834 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:08:39.835 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:08:39.835 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:08:39.835 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:08:39.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:08:39.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:08:39.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:08:39.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:08:39.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:08:39.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:08:39.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:08:39.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:08:39.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:08:39.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:08:39.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:08:39.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:08:39.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:08:39.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-16T08:08:39.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:08:39.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=31369a2e-19cb-454a-8ab1-12b5b7950d31 2024-09-16T08:08:39.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 31369a2e-19cb-454a-8ab1-12b5b7950d31' 2024-09-16T08:08:39.839 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 31369a2e-19cb-454a-8ab1-12b5b7950d31 2024-09-16T08:08:39.839 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:08:39.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCH5+dmd3V8MxAADoWxH4+ApHyiNPysqZkaNw== 2024-09-16T08:08:39.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCH5+dmd3V8MxAADoWxH4+ApHyiNPysqZkaNw=="}' 2024-09-16T08:08:39.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 31369a2e-19cb-454a-8ab1-12b5b7950d31 -i td/test-erasure-eio/0/new.json 2024-09-16T08:08:40.263 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:08:40.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-16T08:08:40.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCH5+dmd3V8MxAADoWxH4+ApHyiNPysqZkaNw== --osd-uuid 31369a2e-19cb-454a-8ab1-12b5b7950d31 2024-09-16T08:08:40.313 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:08:40.311+0000 7f1f8e7b9880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:08:40.316 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:08:40.311+0000 7f1f8e7b9880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:08:40.318 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:08:40.315+0000 7f1f8e7b9880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:08:40.318 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:08:40.315+0000 7f1f8e7b9880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T08:08:40.318 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:08:40.315+0000 7f1f8e7b9880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T08:08:40.319 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:08:40.315+0000 7f1f8e7b9880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T08:08:40.387 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:08:40.383+0000 7f1f8e7b9880 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-16T08:08:46.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-16T08:08:46.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:08:46.125 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T08:08:46.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T08:08:46.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:08:46.694 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T08:08:46.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T08:08:46.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:08:46.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:08:46.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:08:46.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:08:46.730 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:08:46.727+0000 7fd04be0f880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:08:46.776 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:08:46.771+0000 7fd04be0f880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:08:46.778 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:08:46.775+0000 7fd04be0f880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:08:47.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T08:08:47.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:08:47.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T08:08:47.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:08:47.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:08:47.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:08:47.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:08:47.091 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:08:47.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:08:47.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:08:47.348 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:08:47.343+0000 7fd04be0f880 -1 Falling back to public interface 2024-09-16T08:08:47.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:08:48.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:08:48.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:08:48.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:08:48.584 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:08:48.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:08:48.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:08:48.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:08:49.822 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:08:49.819+0000 7fd04be0f880 -1 osd.0 0 log_to_monitors true 2024-09-16T08:08:49.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:08:49.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:08:49.998 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:08:49.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:08:49.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:08:49.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:08:50.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:08:51.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:08:51.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:08:51.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:08:51.723 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:08:51.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:08:51.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:08:52.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:08:53.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:08:53.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:08:53.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:08:53.186 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:08:53.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:08:53.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:08:53.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:08:54.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:08:54.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:08:54.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:08:54.715 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:08:54.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:08:54.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:08:55.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:08:56.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:08:56.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:08:56.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:08:56.164 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:08:56.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:08:56.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:08:56.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:08:57.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:08:57.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:08:57.745 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:08:57.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:08:57.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:08:57.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:08:58.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:08:59.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:08:59.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:08:59.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:08:59.176 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:08:59.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:08:59.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:08:59.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:00.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:00.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:00.682 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:09:00.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:09:00.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:00.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:09:01.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:02.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:02.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:02.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:09:02.101 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:09:02.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:02.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:09:02.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:03.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:03.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:03.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:09:03.533 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:09:03.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:03.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:09:04.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:05.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:05.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:05.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:09:05.411 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:09:05.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:05.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:09:06.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:07.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:07.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:07.496 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:09:07.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:09:07.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:07.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:09:08.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:09.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:09.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:09.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:09:09.828 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:09:09.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:09.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:09:10.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:11.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:11.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:11.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:09:11.515 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:09:11.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:11.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:09:12.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:13.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:13.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:13.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:09:13.371 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:09:13.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:09:13.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:13.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:14.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:14.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:14.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:09:14.905 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:09:14.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:14.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:09:15.372 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3396418464,v1:127.0.0.1:6803/3396418464] [v2:127.0.0.1:6804/3396418464,v1:127.0.0.1:6805/3396418464] exists,up 31369a2e-19cb-454a-8ab1-12b5b7950d31 2024-09-16T08:09:15.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:09:15.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:09:15.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:09:15.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:09:15.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-16T08:09:15.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:09:15.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:09:15.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T08:09:15.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:09:15.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-16T08:09:15.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:09:15.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:09:15.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:09:15.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:09:15.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-16T08:09:15.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-16T08:09:15.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:09:15.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:09:15.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:09:15.378 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:09:15.378 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:09:15.378 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:09:15.378 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:09:15.379 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:09:15.379 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:09:15.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:09:15.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:09:15.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:09:15.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:09:15.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:09:15.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:09:15.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:09:15.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:09:15.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:09:15.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:09:15.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:09:15.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:09:15.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:09:15.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-16T08:09:15.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:09:15.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2e5de68a-2e90-47df-b6bf-074eba0f06d7 2024-09-16T08:09:15.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2e5de68a-2e90-47df-b6bf-074eba0f06d7' 2024-09-16T08:09:15.382 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 2e5de68a-2e90-47df-b6bf-074eba0f06d7 2024-09-16T08:09:15.382 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:09:15.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCr5+dmQ0gSGBAAalGz484/NMJMwYehbqt1rg== 2024-09-16T08:09:15.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCr5+dmQ0gSGBAAalGz484/NMJMwYehbqt1rg=="}' 2024-09-16T08:09:15.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2e5de68a-2e90-47df-b6bf-074eba0f06d7 -i td/test-erasure-eio/1/new.json 2024-09-16T08:09:15.916 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:09:15.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-16T08:09:15.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCr5+dmQ0gSGBAAalGz484/NMJMwYehbqt1rg== --osd-uuid 2e5de68a-2e90-47df-b6bf-074eba0f06d7 2024-09-16T08:09:15.962 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:09:15.959+0000 7f40432f3880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:09:16.008 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:09:16.003+0000 7f40432f3880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:09:16.010 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:09:16.007+0000 7f40432f3880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:09:16.011 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:09:16.007+0000 7f40432f3880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T08:09:16.011 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:09:16.007+0000 7f40432f3880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T08:09:16.011 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:09:16.007+0000 7f40432f3880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T08:09:16.113 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:09:16.111+0000 7f40432f3880 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-16T08:09:22.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-16T08:09:22.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:09:22.088 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-16T08:09:22.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T08:09:22.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:09:22.764 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-16T08:09:22.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T08:09:22.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:09:22.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:09:22.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:09:22.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:09:22.797 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:09:22.790+0000 7f7334e4e880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:09:22.799 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:09:22.794+0000 7f7334e4e880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:09:22.802 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:09:22.798+0000 7f7334e4e880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:09:23.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T08:09:23.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:09:23.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T08:09:23.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:09:23.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:09:23.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:23.175 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:09:23.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:09:23.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:09:23.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:23.372 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:09:23.366+0000 7f7334e4e880 -1 Falling back to public interface 2024-09-16T08:09:23.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:24.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:24.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:24.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:09:24.589 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:09:24.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:24.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:09:25.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:25.709 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:09:25.706+0000 7f7334e4e880 -1 osd.1 0 log_to_monitors true 2024-09-16T08:09:26.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:26.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:26.025 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:09:26.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:09:26.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:26.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:09:26.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:27.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:27.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:27.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:09:27.579 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:09:27.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:27.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:09:28.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:29.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:29.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:29.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:09:29.147 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:09:29.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:29.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:09:29.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:30.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:30.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:30.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:09:30.738 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:09:30.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:30.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:09:31.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:32.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:32.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:32.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:09:32.225 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:09:32.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:32.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:09:32.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:33.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:33.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:33.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:09:33.685 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:09:33.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:33.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:09:34.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:35.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:35.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:35.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:09:35.128 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:09:35.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:35.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:09:35.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:36.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:36.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:36.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:09:36.597 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:09:36.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:36.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:09:37.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:38.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:38.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:38.149 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:09:38.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:09:38.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:38.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:09:38.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:39.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:39.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:39.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:09:39.715 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:09:39.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:39.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:09:40.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:41.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:41.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:41.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:09:41.670 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:09:41.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:41.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:09:42.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:43.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:43.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:43.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:09:43.631 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:09:43.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:43.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:09:44.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:45.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:45.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:45.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:09:45.635 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:09:45.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:45.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:09:46.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:47.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:47.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:47.730 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:09:47.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:09:47.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:47.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:09:48.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:49.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:49.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:49.438 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:09:49.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:09:49.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:49.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:09:50.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:09:51.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:09:51.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:09:51.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:09:51.399 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:09:51.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:09:51.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:09:52.006 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1651060848,v1:127.0.0.1:6811/1651060848] [v2:127.0.0.1:6812/1651060848,v1:127.0.0.1:6813/1651060848] exists,up 2e5de68a-2e90-47df-b6bf-074eba0f06d7 2024-09-16T08:09:52.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:09:52.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:09:52.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:09:52.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:09:52.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-16T08:09:52.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:09:52.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:09:52.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T08:09:52.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:09:52.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-16T08:09:52.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:09:52.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:09:52.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:09:52.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:09:52.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-16T08:09:52.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-16T08:09:52.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:09:52.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:09:52.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:09:52.010 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:09:52.010 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:09:52.010 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:09:52.011 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:09:52.012 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:09:52.012 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:09:52.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:09:52.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:09:52.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:09:52.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:09:52.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:09:52.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:09:52.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:09:52.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:09:52.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:09:52.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:09:52.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:09:52.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:09:52.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:09:52.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-16T08:09:52.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:09:52.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2ef77d20-da35-4cb6-a900-1917cab68221 2024-09-16T08:09:52.014 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 2ef77d20-da35-4cb6-a900-1917cab68221 2024-09-16T08:09:52.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2ef77d20-da35-4cb6-a900-1917cab68221' 2024-09-16T08:09:52.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:09:52.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDQ5+dm0hCPAhAAu8+EhvCVX0dYgp+NBTjdFg== 2024-09-16T08:09:52.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDQ5+dm0hCPAhAAu8+EhvCVX0dYgp+NBTjdFg=="}' 2024-09-16T08:09:52.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2ef77d20-da35-4cb6-a900-1917cab68221 -i td/test-erasure-eio/2/new.json 2024-09-16T08:09:52.577 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:09:52.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-16T08:09:52.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDQ5+dm0hCPAhAAu8+EhvCVX0dYgp+NBTjdFg== --osd-uuid 2ef77d20-da35-4cb6-a900-1917cab68221 2024-09-16T08:09:52.633 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:09:52.630+0000 7f02de9ab880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:09:52.659 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:09:52.658+0000 7f02de9ab880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:09:52.661 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:09:52.658+0000 7f02de9ab880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:09:52.661 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:09:52.662+0000 7f02de9ab880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:09:52.662 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:09:52.662+0000 7f02de9ab880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:09:52.662 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:09:52.662+0000 7f02de9ab880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:09:52.800 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:09:52.798+0000 7f02de9ab880 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-16T08:09:59.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-16T08:09:59.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:09:59.240 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-16T08:09:59.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T08:09:59.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:09:59.820 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-16T08:09:59.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T08:09:59.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:09:59.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:09:59.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:09:59.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:09:59.855 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:09:59.854+0000 7f2fb5dd6880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:09:59.858 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:09:59.858+0000 7f2fb5dd6880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:09:59.860 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:09:59.858+0000 7f2fb5dd6880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:10:00.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T08:10:00.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:10:00.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T08:10:00.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:10:00.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:10:00.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:00.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:10:00.266 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:10:00.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:00.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:10:00.444 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:10:00.442+0000 7f2fb5dd6880 -1 Falling back to public interface 2024-09-16T08:10:00.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:01.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:01.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:01.703 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:10:01.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:10:01.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:01.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:10:02.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:02.982 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:10:02.982+0000 7f2fb5dd6880 -1 osd.2 0 log_to_monitors true 2024-09-16T08:10:03.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:03.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:03.130 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:10:03.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:10:03.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:03.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:10:03.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:04.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:04.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:04.759 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:10:04.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:10:04.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:04.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:10:05.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:06.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:06.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:06.401 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:10:06.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:10:06.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:06.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:10:06.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:07.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:07.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:07.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:10:07.840 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:10:07.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:07.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:10:08.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:09.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:09.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:09.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:10:09.281 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:10:09.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:09.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:10:09.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:10.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:10.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:10.807 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:10:10.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:10:10.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:10.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:10:11.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:12.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:12.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:12.258 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:10:12.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:10:12.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:12.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:10:12.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:13.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:13.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:13.810 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:10:13.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:10:13.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:13.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:10:14.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:15.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:15.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:15.290 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:10:15.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:10:15.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:15.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:10:15.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:16.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:16.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:16.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:10:16.787 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:10:16.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:16.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:10:17.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:18.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:18.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:18.511 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:10:18.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:10:18.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:18.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:10:19.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:20.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:20.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:20.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:10:20.066 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:10:20.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:20.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:10:20.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:21.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:21.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:21.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:10:21.845 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:10:21.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:21.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:10:22.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:23.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:23.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:23.860 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:10:23.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:10:23.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:23.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:10:24.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:25.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:25.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:25.659 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:10:25.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:10:25.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:25.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:10:26.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:27.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:27.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:27.472 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:10:27.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:10:27.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:27.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:10:27.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:28.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:28.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:28.955 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-16T08:10:28.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T08:10:28.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:28.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:10:29.426 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2452318050,v1:127.0.0.1:6819/2452318050] [v2:127.0.0.1:6820/2452318050,v1:127.0.0.1:6821/2452318050] exists,up 2ef77d20-da35-4cb6-a900-1917cab68221 2024-09-16T08:10:29.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:10:29.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:10:29.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:10:29.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:10:29.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-16T08:10:29.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:10:29.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:10:29.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T08:10:29.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:10:29.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-16T08:10:29.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:10:29.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:10:29.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:10:29.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:10:29.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-16T08:10:29.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-16T08:10:29.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:10:29.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:10:29.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:10:29.430 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:10:29.430 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:10:29.430 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:10:29.432 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:10:29.432 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:10:29.432 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:10:29.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:10:29.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:10:29.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:10:29.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:10:29.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:10:29.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:10:29.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:10:29.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:10:29.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:10:29.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:10:29.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:10:29.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:10:29.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:10:29.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-16T08:10:29.435 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:10:29.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=44b0a228-3f30-4ac6-ba25-4769fa3c842f 2024-09-16T08:10:29.435 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 44b0a228-3f30-4ac6-ba25-4769fa3c842f 2024-09-16T08:10:29.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 44b0a228-3f30-4ac6-ba25-4769fa3c842f' 2024-09-16T08:10:29.436 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:10:29.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD15+dmcKKSGxAAnEUTEDFpEin/1O3MAWgimQ== 2024-09-16T08:10:29.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD15+dmcKKSGxAAnEUTEDFpEin/1O3MAWgimQ=="}' 2024-09-16T08:10:29.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 44b0a228-3f30-4ac6-ba25-4769fa3c842f -i td/test-erasure-eio/3/new.json 2024-09-16T08:10:30.752 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:10:30.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-16T08:10:30.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD15+dmcKKSGxAAnEUTEDFpEin/1O3MAWgimQ== --osd-uuid 44b0a228-3f30-4ac6-ba25-4769fa3c842f 2024-09-16T08:10:30.798 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:10:30.797+0000 7f7f92881880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:10:30.884 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:10:30.881+0000 7f7f92881880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:10:30.886 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:10:30.885+0000 7f7f92881880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:10:30.886 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:10:30.885+0000 7f7f92881880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:10:30.886 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:10:30.885+0000 7f7f92881880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:10:30.887 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:10:30.885+0000 7f7f92881880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:10:30.986 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:10:30.985+0000 7f7f92881880 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-16T08:10:37.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-16T08:10:37.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:10:37.288 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-16T08:10:37.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T08:10:37.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:10:37.850 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-16T08:10:37.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T08:10:37.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:10:37.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:10:37.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:10:37.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:10:37.882 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:10:37.881+0000 7f06a85af880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:10:37.905 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:10:37.901+0000 7f06a85af880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:10:37.906 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:10:37.905+0000 7f06a85af880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:10:38.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T08:10:38.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:10:38.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T08:10:38.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:10:38.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:10:38.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:38.277 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:10:38.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:10:38.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:38.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:10:38.472 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:10:38.469+0000 7f06a85af880 -1 Falling back to public interface 2024-09-16T08:10:38.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:39.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:39.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:39.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:10:39.750 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:10:39.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:39.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:10:40.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:41.114 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:10:41.113+0000 7f06a85af880 -1 osd.3 0 log_to_monitors true 2024-09-16T08:10:41.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:41.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:41.186 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:10:41.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:10:41.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:41.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:10:41.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:42.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:42.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:42.728 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:10:42.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:10:42.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:42.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:10:43.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:44.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:44.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:44.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:10:44.321 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:10:44.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:44.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:10:44.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:45.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:45.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:45.762 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:10:45.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:10:45.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:45.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:10:46.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:47.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:47.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:47.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:10:47.336 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:10:47.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:47.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:10:47.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:48.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:48.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:48.779 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:10:48.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:10:48.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:48.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:10:49.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:50.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:50.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:50.228 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:10:50.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:10:50.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:50.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:10:50.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:51.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:51.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:51.730 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:10:51.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:10:51.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:51.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:10:52.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:53.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:53.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:53.261 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:10:53.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:10:53.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:53.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:10:53.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:54.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:54.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:54.695 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:10:54.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:10:54.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:54.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:10:55.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:56.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:56.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:56.317 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:10:56.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:10:56.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:56.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:10:57.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:58.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:58.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:58.114 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:10:58.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:10:58.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:58.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:10:58.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:10:59.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:10:59.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:10:59.623 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:10:59.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:10:59.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:10:59.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:11:00.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:11:01.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:11:01.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:11:01.760 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:11:01.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:11:01.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:11:01.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:11:02.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:11:03.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:11:03.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:11:03.671 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:11:03.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:11:03.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:11:03.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:11:04.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:11:05.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:11:05.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:11:05.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:11:05.224 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:11:05.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:11:05.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:11:06.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:11:07.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:11:07.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:11:07.260 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-16T08:11:07.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T08:11:07.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:11:07.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:11:07.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:11:08.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:11:08.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:11:08.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T08:11:08.699 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-09-16T08:11:08.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:11:08.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:11:09.222 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1015127740,v1:127.0.0.1:6827/1015127740] [v2:127.0.0.1:6828/1015127740,v1:127.0.0.1:6829/1015127740] exists,up 44b0a228-3f30-4ac6-ba25-4769fa3c842f 2024-09-16T08:11:09.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:11:09.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:11:09.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:11:09.224 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-16T08:11:09.224 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T08:11:09.224 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T08:11:09.224 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:11:09.225 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:11:09.225 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:11:09.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.0.asok 2024-09-16T08:11:09.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-16T08:11:09.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.0.asok log flush 2024-09-16T08:11:09.310 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T08:11:09.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-16T08:11:09.325 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T08:08:47.355+0000 7fd04be0f880 0 load: jerasure load: lrc 2024-09-16T08:11:09.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:305: TEST_rados_get_bad_size_shard_0: local poolname=pool-jerasure 2024-09-16T08:11:09.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:306: TEST_rados_get_bad_size_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2024-09-16T08:11:09.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T08:11:09.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-16T08:11:09.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-16T08:11:09.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-16T08:11:09.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-16T08:11:09.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-16T08:11:09.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-16T08:11:10.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-16T08:11:10.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-16T08:11:11.333 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-16T08:11:11.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T08:11:12.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-16T08:11:12.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:11:12.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:11:12.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:11:12.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:11:12.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:11:12.347 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:11:12.347 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:11:12.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:11:12.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:11:12.348 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:11:12.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:11:12.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:11:12.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:11:12.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:11:12.567 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:11:13.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:11:13.047 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:11:13.047 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:11:13.047 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:11:13.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:11:13.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:11:13.048 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:11:13.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803815 2024-09-16T08:11:13.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803815 2024-09-16T08:11:13.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803815' 2024-09-16T08:11:13.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:11:13.278 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:11:13.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607586 2024-09-16T08:11:13.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607586 2024-09-16T08:11:13.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803815 1-51539607586' 2024-09-16T08:11:13.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:11:13.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:11:13.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=77309411354 2024-09-16T08:11:13.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 77309411354 2024-09-16T08:11:13.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803815 1-51539607586 2-77309411354' 2024-09-16T08:11:13.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:11:13.591 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:11:13.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=120259084305 2024-09-16T08:11:13.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 120259084305 2024-09-16T08:11:13.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803815 1-51539607586 2-77309411354 3-120259084305' 2024-09-16T08:11:13.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:11:13.741 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803815 2024-09-16T08:11:13.741 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:11:13.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:11:13.744 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803815 2024-09-16T08:11:13.745 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:11:13.747 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803815 2024-09-16T08:11:13.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803815 2024-09-16T08:11:13.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803815' 2024-09-16T08:11:13.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:11:14.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803815 -lt 25769803815 2024-09-16T08:11:14.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:11:14.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607586 2024-09-16T08:11:14.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:11:14.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:11:14.179 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607586 2024-09-16T08:11:14.179 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:11:14.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607586 2024-09-16T08:11:14.181 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 51539607586 2024-09-16T08:11:14.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607586' 2024-09-16T08:11:14.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:11:14.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607586 -lt 51539607586 2024-09-16T08:11:14.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:11:14.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-77309411354 2024-09-16T08:11:14.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:11:14.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:11:14.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-77309411354 2024-09-16T08:11:14.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:11:14.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=77309411354 2024-09-16T08:11:14.686 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 77309411354 2024-09-16T08:11:14.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 77309411354' 2024-09-16T08:11:14.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:11:15.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 77309411354 -lt 77309411354 2024-09-16T08:11:15.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:11:15.134 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-120259084305 2024-09-16T08:11:15.134 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:11:15.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:11:15.137 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:11:15.137 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-120259084305 2024-09-16T08:11:15.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=120259084305 2024-09-16T08:11:15.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 120259084305' 2024-09-16T08:11:15.138 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 120259084305 2024-09-16T08:11:15.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:11:15.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 120259084305 -lt 120259084305 2024-09-16T08:11:15.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:11:15.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:11:15.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:11:16.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:11:16.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:11:16.180 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:11:16.180 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:11:16.180 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:11:16.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:11:16.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:11:16.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:11:16.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:11:16.584 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:11:16.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:11:16.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:11:17.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:11:17.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:11:17.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:11:17.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:308: TEST_rados_get_bad_size_shard_0: local shard_id=0 2024-09-16T08:11:17.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:309: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 10 2024-09-16T08:11:17.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-16T08:11:17.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-09-16T08:11:17.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=0 2024-09-16T08:11:17.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-09-16T08:11:17.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=10 2024-09-16T08:11:17.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-09-16T08:11:17.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-09-16T08:11:17.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-16T08:11:17.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-79228-0-10 2024-09-16T08:11:17.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-79228-0-10 2024-09-16T08:11:17.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-16T08:11:17.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-16T08:11:17.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-79228-0-10 2024-09-16T08:11:17.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:11:17.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-16T08:11:17.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:11:17.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-16T08:11:17.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:11:17.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-16T08:11:17.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:11:17.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-16T08:11:17.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-79228-0-10 td/test-erasure-eio/ORIGINAL 2024-09-16T08:11:17.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-79228-0-10 td/test-erasure-eio 0 10 set 2024-09-16T08:11:17.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79228-0-10 2024-09-16T08:11:17.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-16T08:11:17.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-16T08:11:17.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-16T08:11:17.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=0 2024-09-16T08:11:17.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-16T08:11:17.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-09-16T08:11:17.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-16T08:11:17.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-09-16T08:11:17.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-16T08:11:17.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:11:17.356 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79228-0-10 2024-09-16T08:11:17.356 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:11:17.356 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79228-0-10 2024-09-16T08:11:17.356 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79228-0-10 2024-09-16T08:11:17.357 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:11:17.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:11:17.822 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:11:17.822 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T08:11:17.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-16T08:11:17.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-16T08:11:17.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=3 2024-09-16T08:11:17.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-16T08:11:18.732 INFO:tasks.workunit.client.0.smithi162.stderr:noout is set 2024-09-16T08:11:18.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-09-16T08:11:18.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-09-16T08:11:18.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-09-16T08:11:18.748 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records in 2024-09-16T08:11:18.748 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records out 2024-09-16T08:11:18.748 INFO:tasks.workunit.client.0.smithi162.stderr:10 bytes copied, 0.000209303 s, 47.8 kB/s 2024-09-16T08:11:18.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-79228-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:11:18.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-16T08:11:18.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T08:11:18.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-16T08:11:18.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T08:11:18.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-79228-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:11:18.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-16T08:11:18.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T08:11:18.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-16T08:11:18.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T08:11:18.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-16T08:11:18.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:11:18.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:11:18.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:11:18.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:11:18.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:11:20.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:11:20.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-79228-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:11:20.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-16T08:11:20.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T08:11:20.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-16T08:11:20.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T08:11:20.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-09-16T08:11:20.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-79228-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:11:22.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-09-16T08:11:22.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-16T08:11:22.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T08:11:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-16T08:11:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T08:11:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-16T08:11:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:11:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:11:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:11:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:11:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-16T08:11:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-16T08:11:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T08:11:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T08:11:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:11:22.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T08:11:22.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:11:22.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:11:22.038 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:11:22.038 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:11:22.039 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:11:22.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:11:22.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:11:22.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:11:22.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:11:22.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:11:22.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:11:22.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:11:22.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:11:22.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:11:22.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T08:11:22.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T08:11:22.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-16T08:11:22.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-16T08:11:22.040 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.3 2024-09-16T08:11:22.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:11:22.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-16T08:11:22.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-16T08:11:22.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T08:11:22.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T08:11:22.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T08:11:22.076 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:11:22.072+0000 7f74d2da6880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:11:22.079 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:11:22.076+0000 7f74d2da6880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:11:22.081 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:11:22.080+0000 7f74d2da6880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:11:22.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-16T08:11:22.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:11:22.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T08:11:22.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:11:22.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:11:22.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:11:22.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:11:22.469 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T08:11:22.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:11:22.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:11:22.628 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:11:22.624+0000 7f74d2da6880 -1 Falling back to public interface 2024-09-16T08:11:22.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:11:23.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:11:23.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:11:23.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:11:23.975 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:11:23.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:11:23.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:11:24.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:11:25.203 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:11:25.200+0000 7f74d2da6880 -1 osd.3 37 log_to_monitors true 2024-09-16T08:11:25.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:11:25.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:11:25.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:11:25.402 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:11:25.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:11:25.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:11:26.024 INFO:tasks.workunit.client.0.smithi162.stderr:osd.3 up in weight 1 up_from 42 up_thru 34 down_at 39 last_clean_interval [28,37) [v2:127.0.0.1:6826/3477188465,v1:127.0.0.1:6827/3477188465] [v2:127.0.0.1:6828/3477188465,v1:127.0.0.1:6829/3477188465] exists,up 44b0a228-3f30-4ac6-ba25-4769fa3c842f 2024-09-16T08:11:26.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:11:26.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:11:26.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:11:26.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-16T08:11:26.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:11:26.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:11:26.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:11:26.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:11:26.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:11:26.026 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:11:26.026 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:11:26.027 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:11:26.027 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:11:26.027 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:11:26.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:11:26.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:11:26.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:11:26.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:11:26.237 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:11:27.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:11:27.095 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:11:27.095 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:11:27.095 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:11:27.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:11:27.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:11:27.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:11:27.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803820 2024-09-16T08:11:27.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803820 2024-09-16T08:11:27.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803820' 2024-09-16T08:11:27.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:11:27.250 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:11:27.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=51539607591 2024-09-16T08:11:27.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 51539607591 2024-09-16T08:11:27.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803820 1-51539607591' 2024-09-16T08:11:27.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:11:27.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:11:27.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=77309411358 2024-09-16T08:11:27.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 77309411358 2024-09-16T08:11:27.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803820 1-51539607591 2-77309411358' 2024-09-16T08:11:27.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:11:27.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:11:27.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=180388626435 2024-09-16T08:11:27.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 180388626435 2024-09-16T08:11:27.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803820 1-51539607591 2-77309411358 3-180388626435' 2024-09-16T08:11:27.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:11:27.708 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803820 2024-09-16T08:11:27.708 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:11:27.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:11:27.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803820 2024-09-16T08:11:27.712 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:11:27.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803820 2024-09-16T08:11:27.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803820' 2024-09-16T08:11:27.713 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803820 2024-09-16T08:11:27.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:11:28.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803820 -lt 25769803820 2024-09-16T08:11:28.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:11:28.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-51539607591 2024-09-16T08:11:28.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:11:28.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:11:28.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-51539607591 2024-09-16T08:11:28.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:11:28.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=51539607591 2024-09-16T08:11:28.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 51539607591' 2024-09-16T08:11:28.193 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 51539607591 2024-09-16T08:11:28.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:11:28.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 51539607591 -lt 51539607591 2024-09-16T08:11:28.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:11:28.679 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-77309411358 2024-09-16T08:11:28.679 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:11:28.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:11:28.682 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-77309411358 2024-09-16T08:11:28.682 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:11:28.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=77309411358 2024-09-16T08:11:28.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 77309411358' 2024-09-16T08:11:28.685 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 77309411358 2024-09-16T08:11:28.685 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:11:29.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 77309411358 -lt 77309411358 2024-09-16T08:11:29.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:11:29.133 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-180388626435 2024-09-16T08:11:29.133 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:11:29.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:11:29.136 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-180388626435 2024-09-16T08:11:29.136 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:11:29.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=180388626435 2024-09-16T08:11:29.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 180388626435' 2024-09-16T08:11:29.138 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 180388626435 2024-09-16T08:11:29.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:11:29.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 180388626435 -lt 180388626435 2024-09-16T08:11:29.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:11:29.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:11:29.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:11:30.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:11:30.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:11:30.163 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:11:30.163 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:11:30.163 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:11:30.163 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:11:30.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:11:30.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:11:30.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:11:30.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:11:30.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:11:30.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:11:31.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:11:31.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:11:31.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:11:31.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-16T08:11:31.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-16T08:11:31.953 INFO:tasks.workunit.client.0.smithi162.stderr:noout is unset 2024-09-16T08:11:31.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79228-0-10 2024-09-16T08:11:31.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T08:11:31.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T08:11:31.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79228-0-10 2024-09-16T08:11:31.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-16T08:11:31.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-16T08:11:31.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-79228-0-10 td/test-erasure-eio/COPY 2024-09-16T08:11:32.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-16T08:11:32.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-16T08:11:32.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 0 + 1 2024-09-16T08:11:32.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=1 2024-09-16T08:11:32.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-79228-0-10 td/test-erasure-eio 1 10 set 2024-09-16T08:11:32.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79228-0-10 2024-09-16T08:11:32.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-16T08:11:32.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-16T08:11:32.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-16T08:11:32.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-09-16T08:11:32.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-16T08:11:32.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-09-16T08:11:32.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-16T08:11:32.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-09-16T08:11:32.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-16T08:11:32.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:11:32.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79228-0-10 2024-09-16T08:11:32.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:11:32.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79228-0-10 2024-09-16T08:11:32.148 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79228-0-10 2024-09-16T08:11:32.148 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:11:32.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:11:32.659 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:11:32.659 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T08:11:32.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-16T08:11:32.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-16T08:11:32.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-09-16T08:11:32.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-16T08:11:33.417 INFO:tasks.workunit.client.0.smithi162.stderr:noout is set 2024-09-16T08:11:33.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-09-16T08:11:33.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-09-16T08:11:33.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-09-16T08:11:33.435 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records in 2024-09-16T08:11:33.436 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records out 2024-09-16T08:11:33.436 INFO:tasks.workunit.client.0.smithi162.stderr:10 bytes copied, 0.00019644 s, 50.9 kB/s 2024-09-16T08:11:33.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-79228-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:11:33.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-16T08:11:33.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T08:11:33.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-16T08:11:33.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T08:11:33.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-79228-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:11:33.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-16T08:11:33.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T08:11:33.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-16T08:11:33.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T08:11:33.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-16T08:11:33.437 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:11:33.437 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:11:33.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:11:33.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:11:33.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:11:34.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:11:34.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-79228-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:11:34.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-16T08:11:34.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T08:11:34.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-16T08:11:34.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T08:11:34.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-16T08:11:34.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-79228-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:11:37.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-16T08:11:37.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-16T08:11:37.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T08:11:37.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T08:11:37.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T08:11:37.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-16T08:11:37.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:11:37.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:11:37.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:11:37.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:11:37.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-16T08:11:37.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-16T08:11:37.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T08:11:37.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T08:11:37.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:11:37.005 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T08:11:37.005 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:11:37.005 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:11:37.006 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:11:37.006 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:11:37.006 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:11:37.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:11:37.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:11:37.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:11:37.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:11:37.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:11:37.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:11:37.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:11:37.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:11:37.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:11:37.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T08:11:37.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T08:11:37.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-16T08:11:37.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T08:11:37.008 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-09-16T08:11:37.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:11:37.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-16T08:11:37.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T08:11:37.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T08:11:37.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T08:11:37.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T08:11:37.046 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:11:37.044+0000 7f6784661880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:11:37.050 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:11:37.048+0000 7f6784661880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:11:37.052 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:11:37.048+0000 7f6784661880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:11:37.340 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:11:37.336+0000 7f6784661880 -1 Falling back to public interface 2024-09-16T08:11:37.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T08:11:37.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:11:37.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T08:11:37.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:11:37.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:11:37.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:11:37.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:11:37.447 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T08:11:37.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:11:37.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:11:37.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:11:38.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:11:38.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:11:38.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:11:38.875 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:11:38.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:11:38.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:11:39.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:11:40.040 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:11:40.036+0000 7f6784661880 -1 osd.1 48 log_to_monitors true 2024-09-16T08:11:40.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:11:40.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:11:40.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:11:40.307 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:11:40.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:11:40.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:11:40.987 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 53 up_thru 39 down_at 49 last_clean_interval [12,48) [v2:127.0.0.1:6810/2108923736,v1:127.0.0.1:6811/2108923736] [v2:127.0.0.1:6812/2108923736,v1:127.0.0.1:6813/2108923736] exists,up 2e5de68a-2e90-47df-b6bf-074eba0f06d7 2024-09-16T08:11:40.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:11:40.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:11:40.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:11:40.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-16T08:11:40.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:11:40.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:11:40.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:11:40.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:11:40.989 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:11:40.989 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:11:40.989 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:11:40.989 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:11:40.989 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:11:40.989 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:11:41.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:11:41.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:11:41.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:11:41.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:11:41.194 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:11:41.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:11:41.796 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:11:41.797 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:11:41.797 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:11:41.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:11:41.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:11:41.798 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:11:41.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803825 2024-09-16T08:11:41.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803825 2024-09-16T08:11:41.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803825' 2024-09-16T08:11:41.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:11:41.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:11:42.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=227633266691 2024-09-16T08:11:42.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 227633266691 2024-09-16T08:11:42.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803825 1-227633266691' 2024-09-16T08:11:42.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:11:42.213 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:11:42.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=77309411363 2024-09-16T08:11:42.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 77309411363 2024-09-16T08:11:42.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803825 1-227633266691 2-77309411363' 2024-09-16T08:11:42.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:11:42.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:11:42.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=180388626440 2024-09-16T08:11:42.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 180388626440 2024-09-16T08:11:42.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803825 1-227633266691 2-77309411363 3-180388626440' 2024-09-16T08:11:42.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:11:42.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803825 2024-09-16T08:11:42.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:11:42.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:11:42.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803825 2024-09-16T08:11:42.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:11:42.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803825 2024-09-16T08:11:42.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803825' 2024-09-16T08:11:42.563 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803825 2024-09-16T08:11:42.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:11:42.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803823 -lt 25769803825 2024-09-16T08:11:42.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T08:11:43.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T08:11:43.993 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:11:44.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803825 -lt 25769803825 2024-09-16T08:11:44.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:11:44.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-227633266691 2024-09-16T08:11:44.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:11:44.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:11:44.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-227633266691 2024-09-16T08:11:44.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:11:44.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=227633266691 2024-09-16T08:11:44.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 227633266691' 2024-09-16T08:11:44.451 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 227633266691 2024-09-16T08:11:44.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:11:44.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 227633266691 -lt 227633266691 2024-09-16T08:11:44.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:11:44.894 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:11:44.895 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-77309411363 2024-09-16T08:11:44.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:11:44.897 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-77309411363 2024-09-16T08:11:44.897 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:11:44.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=77309411363 2024-09-16T08:11:44.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 77309411363' 2024-09-16T08:11:44.900 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 77309411363 2024-09-16T08:11:44.900 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:11:45.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 77309411364 -lt 77309411363 2024-09-16T08:11:45.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:11:45.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-180388626440 2024-09-16T08:11:45.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:11:45.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:11:45.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-180388626440 2024-09-16T08:11:45.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:11:45.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=180388626440 2024-09-16T08:11:45.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 180388626440' 2024-09-16T08:11:45.355 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 180388626440 2024-09-16T08:11:45.356 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:11:45.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 180388626440 -lt 180388626440 2024-09-16T08:11:45.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:11:45.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:11:45.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:11:46.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:11:46.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:11:46.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:11:46.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:11:46.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:11:46.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:11:46.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:11:46.372 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:11:46.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:11:46.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:11:46.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:11:46.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:11:47.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:11:47.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:11:47.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:11:47.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-16T08:11:47.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-16T08:11:48.362 INFO:tasks.workunit.client.0.smithi162.stderr:noout is unset 2024-09-16T08:11:48.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79228-0-10 fail 2024-09-16T08:11:48.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T08:11:48.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T08:11:48.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79228-0-10 2024-09-16T08:11:48.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-16T08:11:48.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-16T08:11:48.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-79228-0-10 td/test-erasure-eio/COPY 2024-09-16T08:11:48.504 INFO:tasks.workunit.client.0.smithi162.stderr:error getting pool-jerasure/obj-size-79228-0-10: (5) Input/output error 2024-09-16T08:11:48.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-16T08:11:48.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-16T08:11:48.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:310: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 0 2024-09-16T08:11:48.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-16T08:11:48.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-09-16T08:11:48.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=0 2024-09-16T08:11:48.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-09-16T08:11:48.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=0 2024-09-16T08:11:48.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-09-16T08:11:48.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-09-16T08:11:48.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-16T08:11:48.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-79228-0-0 2024-09-16T08:11:48.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-79228-0-0 2024-09-16T08:11:48.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-16T08:11:48.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-16T08:11:48.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-79228-0-0 2024-09-16T08:11:48.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:11:48.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-16T08:11:48.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:11:48.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-16T08:11:48.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:11:48.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-16T08:11:48.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:11:48.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-16T08:11:48.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-79228-0-0 td/test-erasure-eio/ORIGINAL 2024-09-16T08:11:48.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-79228-0-0 td/test-erasure-eio 0 0 set 2024-09-16T08:11:48.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79228-0-0 2024-09-16T08:11:48.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-16T08:11:48.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-16T08:11:48.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-16T08:11:48.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=0 2024-09-16T08:11:48.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-16T08:11:48.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2024-09-16T08:11:48.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-16T08:11:48.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-09-16T08:11:48.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-16T08:11:48.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:11:48.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79228-0-0 2024-09-16T08:11:48.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:11:48.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79228-0-0 2024-09-16T08:11:48.712 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79228-0-0 2024-09-16T08:11:48.712 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:11:49.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:11:49.184 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:11:49.184 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T08:11:49.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-16T08:11:49.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-16T08:11:49.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=3 2024-09-16T08:11:49.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-16T08:11:50.217 INFO:tasks.workunit.client.0.smithi162.stderr:noout is set 2024-09-16T08:11:50.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-09-16T08:11:50.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2024-09-16T08:11:50.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2024-09-16T08:11:50.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-79228-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:11:50.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-16T08:11:50.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T08:11:50.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-16T08:11:50.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T08:11:50.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-79228-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:11:50.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-16T08:11:50.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T08:11:50.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-16T08:11:50.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T08:11:50.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-16T08:11:50.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:11:50.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:11:50.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:11:50.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:11:50.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:11:50.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:11:50.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-79228-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:11:50.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-16T08:11:50.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T08:11:50.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-16T08:11:50.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T08:11:50.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-09-16T08:11:50.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-79228-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:11:53.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-09-16T08:11:53.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-16T08:11:53.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T08:11:53.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-16T08:11:53.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T08:11:53.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-16T08:11:53.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:11:53.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:11:53.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:11:53.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:11:53.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-16T08:11:53.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-16T08:11:53.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T08:11:53.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T08:11:53.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:11:53.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T08:11:53.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:11:53.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:11:53.250 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:11:53.250 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:11:53.250 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:11:53.250 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:11:53.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:11:53.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:11:53.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:11:53.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:11:53.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:11:53.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:11:53.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:11:53.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:11:53.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T08:11:53.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T08:11:53.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-16T08:11:53.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-16T08:11:53.252 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.3 2024-09-16T08:11:53.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:11:53.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-16T08:11:53.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-16T08:11:53.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T08:11:53.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T08:11:53.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T08:11:53.288 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:11:53.283+0000 7f66def6a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:11:53.291 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:11:53.287+0000 7f66def6a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:11:53.293 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:11:53.291+0000 7f66def6a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:11:53.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-16T08:11:53.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:11:53.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T08:11:53.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:11:53.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:11:53.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:11:53.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:11:53.697 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T08:11:53.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:11:53.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:11:53.839 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:11:53.835+0000 7f66def6a880 -1 Falling back to public interface 2024-09-16T08:11:54.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:11:55.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:11:55.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:11:55.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:11:55.127 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:11:55.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:11:55.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:11:55.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:11:56.374 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:11:56.371+0000 7f66def6a880 -1 osd.3 58 log_to_monitors true 2024-09-16T08:11:56.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:11:56.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:11:56.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:11:56.544 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:11:56.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:11:56.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:11:57.121 INFO:tasks.workunit.client.0.smithi162.stderr:osd.3 up in weight 1 up_from 63 up_thru 53 down_at 59 last_clean_interval [42,58) [v2:127.0.0.1:6826/1112870093,v1:127.0.0.1:6827/1112870093] [v2:127.0.0.1:6828/1112870093,v1:127.0.0.1:6829/1112870093] exists,up 44b0a228-3f30-4ac6-ba25-4769fa3c842f 2024-09-16T08:11:57.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:11:57.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:11:57.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:11:57.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-16T08:11:57.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:11:57.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:11:57.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:11:57.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:11:57.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:11:57.123 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:11:57.123 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:11:57.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:11:57.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:11:57.124 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:11:57.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:11:57.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:11:57.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:11:57.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:11:57.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:11:57.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:11:57.958 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:11:57.958 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:11:57.958 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:11:57.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:11:57.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:11:57.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:11:58.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803830 2024-09-16T08:11:58.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803830 2024-09-16T08:11:58.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803830' 2024-09-16T08:11:58.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:11:58.215 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:11:58.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=227633266696 2024-09-16T08:11:58.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 227633266696 2024-09-16T08:11:58.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803830 1-227633266696' 2024-09-16T08:11:58.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:11:58.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:11:58.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=77309411369 2024-09-16T08:11:58.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 77309411369 2024-09-16T08:11:58.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803830 1-227633266696 2-77309411369' 2024-09-16T08:11:58.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:11:58.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:11:58.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=270582939651 2024-09-16T08:11:58.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 270582939651 2024-09-16T08:11:58.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803830 1-227633266696 2-77309411369 3-270582939651' 2024-09-16T08:11:58.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:11:58.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803830 2024-09-16T08:11:58.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:11:58.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:11:58.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803830 2024-09-16T08:11:58.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:11:58.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803830 2024-09-16T08:11:58.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803830' 2024-09-16T08:11:58.756 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803830 2024-09-16T08:11:58.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:11:59.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803828 -lt 25769803830 2024-09-16T08:11:59.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T08:12:00.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T08:12:00.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:12:00.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803830 -lt 25769803830 2024-09-16T08:12:00.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:12:00.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-227633266696 2024-09-16T08:12:00.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:12:00.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:12:00.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-227633266696 2024-09-16T08:12:00.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:12:00.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=227633266696 2024-09-16T08:12:00.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 227633266696' 2024-09-16T08:12:00.631 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 227633266696 2024-09-16T08:12:00.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:12:01.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 227633266696 -lt 227633266696 2024-09-16T08:12:01.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:12:01.032 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-77309411369 2024-09-16T08:12:01.032 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:12:01.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:12:01.035 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-77309411369 2024-09-16T08:12:01.036 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:12:01.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=77309411369 2024-09-16T08:12:01.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 77309411369' 2024-09-16T08:12:01.038 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 77309411369 2024-09-16T08:12:01.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:12:01.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 77309411369 -lt 77309411369 2024-09-16T08:12:01.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:12:01.434 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-270582939651 2024-09-16T08:12:01.434 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:12:01.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:12:01.437 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:12:01.437 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-270582939651 2024-09-16T08:12:01.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=270582939651 2024-09-16T08:12:01.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 270582939651' 2024-09-16T08:12:01.438 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 270582939651 2024-09-16T08:12:01.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:12:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 270582939651 -lt 270582939651 2024-09-16T08:12:01.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:12:01.985 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:12:01.985 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:12:02.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:12:02.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:12:02.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:12:02.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:12:02.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:12:02.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:12:02.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:12:02.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:12:03.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:12:03.005 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:12:03.005 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:12:03.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:12:03.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:12:03.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:12:03.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:12:03.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-16T08:12:03.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-16T08:12:04.434 INFO:tasks.workunit.client.0.smithi162.stderr:noout is unset 2024-09-16T08:12:04.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79228-0-0 2024-09-16T08:12:04.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T08:12:04.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T08:12:04.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79228-0-0 2024-09-16T08:12:04.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-16T08:12:04.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-16T08:12:04.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-79228-0-0 td/test-erasure-eio/COPY 2024-09-16T08:12:04.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-16T08:12:04.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-16T08:12:04.679 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 0 + 1 2024-09-16T08:12:04.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=1 2024-09-16T08:12:04.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-79228-0-0 td/test-erasure-eio 1 0 set 2024-09-16T08:12:04.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79228-0-0 2024-09-16T08:12:04.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-16T08:12:04.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-16T08:12:04.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-16T08:12:04.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-09-16T08:12:04.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-16T08:12:04.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2024-09-16T08:12:04.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-16T08:12:04.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-09-16T08:12:04.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-16T08:12:04.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:12:04.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79228-0-0 2024-09-16T08:12:04.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:12:04.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79228-0-0 2024-09-16T08:12:04.684 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79228-0-0 2024-09-16T08:12:04.684 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:12:05.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:12:05.120 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:12:05.120 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T08:12:05.120 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-16T08:12:05.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-16T08:12:05.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-09-16T08:12:05.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-16T08:12:05.889 INFO:tasks.workunit.client.0.smithi162.stderr:noout is set 2024-09-16T08:12:05.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-09-16T08:12:05.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2024-09-16T08:12:05.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2024-09-16T08:12:05.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-79228-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:12:05.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-16T08:12:05.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T08:12:05.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-16T08:12:05.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T08:12:05.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-79228-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:12:05.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-16T08:12:05.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T08:12:05.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-16T08:12:05.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T08:12:05.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-16T08:12:05.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:12:05.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:12:05.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:12:05.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:12:05.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:12:06.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:12:06.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-79228-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:12:06.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-16T08:12:06.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T08:12:06.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-16T08:12:06.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T08:12:06.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-16T08:12:06.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-79228-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:12:08.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-16T08:12:08.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-16T08:12:08.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T08:12:08.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T08:12:08.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T08:12:08.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-16T08:12:08.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:12:08.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:12:08.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:12:08.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:12:08.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-16T08:12:08.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-16T08:12:08.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T08:12:08.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T08:12:08.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:12:08.852 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T08:12:08.852 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:12:08.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:12:08.854 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:12:08.854 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:12:08.854 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:12:08.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:12:08.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:12:08.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:12:08.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:12:08.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:12:08.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:12:08.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:12:08.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:12:08.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:12:08.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T08:12:08.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T08:12:08.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-16T08:12:08.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T08:12:08.856 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-09-16T08:12:08.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:12:08.857 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-16T08:12:08.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T08:12:08.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T08:12:08.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T08:12:08.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T08:12:08.890 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:12:08.887+0000 7f54f6f70880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:12:08.894 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:12:08.891+0000 7f54f6f70880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:12:08.896 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:12:08.891+0000 7f54f6f70880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:12:09.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T08:12:09.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:12:09.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T08:12:09.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:12:09.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:12:09.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:12:09.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:12:09.332 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T08:12:09.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:12:09.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:12:09.456 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:12:09.451+0000 7f54f6f70880 -1 Falling back to public interface 2024-09-16T08:12:09.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:12:10.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:12:10.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:12:10.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:12:10.833 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:12:10.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:12:10.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:12:11.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:12:12.200 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:12:12.195+0000 7f54f6f70880 -1 osd.1 67 log_to_monitors true 2024-09-16T08:12:12.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:12:12.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:12:12.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:12:12.233 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:12:12.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:12:12.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:12:12.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:12:13.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:12:13.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:12:13.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:12:13.773 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T08:12:13.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:12:13.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:12:14.233 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 74 up_thru 74 down_at 70 last_clean_interval [53,67) [v2:127.0.0.1:6810/193990870,v1:127.0.0.1:6811/193990870] [v2:127.0.0.1:6812/193990870,v1:127.0.0.1:6813/193990870] exists,up 2e5de68a-2e90-47df-b6bf-074eba0f06d7 2024-09-16T08:12:14.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:12:14.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:12:14.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:12:14.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-16T08:12:14.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:12:14.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:12:14.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:12:14.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:12:14.234 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:12:14.235 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:12:14.235 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:12:14.236 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:12:14.236 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:12:14.236 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:12:14.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:12:14.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:12:14.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:12:14.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:12:14.459 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:12:14.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:12:14.926 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:12:14.926 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:12:14.926 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:12:14.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:12:14.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:12:14.926 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:12:15.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803835 2024-09-16T08:12:15.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803835 2024-09-16T08:12:15.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803835' 2024-09-16T08:12:15.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:12:15.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:12:15.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=317827579907 2024-09-16T08:12:15.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 317827579907 2024-09-16T08:12:15.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803835 1-317827579907' 2024-09-16T08:12:15.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:12:15.246 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:12:15.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=77309411374 2024-09-16T08:12:15.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 77309411374 2024-09-16T08:12:15.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803835 1-317827579907 2-77309411374' 2024-09-16T08:12:15.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:12:15.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:12:15.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=270582939656 2024-09-16T08:12:15.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 270582939656 2024-09-16T08:12:15.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803835 1-317827579907 2-77309411374 3-270582939656' 2024-09-16T08:12:15.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:12:15.561 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803835 2024-09-16T08:12:15.561 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:12:15.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:12:15.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803835 2024-09-16T08:12:15.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:12:15.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803835 2024-09-16T08:12:15.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803835' 2024-09-16T08:12:15.566 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803835 2024-09-16T08:12:15.567 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:12:16.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803833 -lt 25769803835 2024-09-16T08:12:16.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T08:12:17.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T08:12:17.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:12:17.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803836 -lt 25769803835 2024-09-16T08:12:17.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:12:17.472 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-317827579907 2024-09-16T08:12:17.472 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:12:17.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:12:17.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-317827579907 2024-09-16T08:12:17.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:12:17.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=317827579907 2024-09-16T08:12:17.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 317827579907' 2024-09-16T08:12:17.476 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 317827579907 2024-09-16T08:12:17.477 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:12:17.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 317827579907 -lt 317827579907 2024-09-16T08:12:17.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:12:17.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-77309411374 2024-09-16T08:12:17.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:12:17.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:12:17.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-77309411374 2024-09-16T08:12:17.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:12:17.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=77309411374 2024-09-16T08:12:17.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 77309411374' 2024-09-16T08:12:17.917 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 77309411374 2024-09-16T08:12:17.917 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:12:18.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 77309411374 -lt 77309411374 2024-09-16T08:12:18.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:12:18.341 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-270582939656 2024-09-16T08:12:18.341 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:12:18.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:12:18.344 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-270582939656 2024-09-16T08:12:18.344 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:12:18.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=270582939656 2024-09-16T08:12:18.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 270582939656' 2024-09-16T08:12:18.347 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 270582939656 2024-09-16T08:12:18.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:12:18.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 270582939657 -lt 270582939656 2024-09-16T08:12:18.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:12:18.851 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:12:18.851 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:12:19.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:12:19.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:12:19.432 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:12:19.432 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:12:19.432 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:12:19.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:12:19.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:12:19.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:12:19.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:12:19.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:12:19.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:12:19.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:12:20.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:12:20.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:12:20.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:12:20.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-16T08:12:20.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-16T08:12:21.206 INFO:tasks.workunit.client.0.smithi162.stderr:noout is unset 2024-09-16T08:12:21.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79228-0-0 fail 2024-09-16T08:12:21.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T08:12:21.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T08:12:21.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79228-0-0 2024-09-16T08:12:21.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-16T08:12:21.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-16T08:12:21.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-79228-0-0 td/test-erasure-eio/COPY 2024-09-16T08:12:21.343 INFO:tasks.workunit.client.0.smithi162.stderr:error getting pool-jerasure/obj-size-79228-0-0: (5) Input/output error 2024-09-16T08:12:21.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-16T08:12:21.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-16T08:12:21.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:311: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 256 add 2024-09-16T08:12:21.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-16T08:12:21.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-09-16T08:12:21.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=0 2024-09-16T08:12:21.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-09-16T08:12:21.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=256 2024-09-16T08:12:21.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-09-16T08:12:21.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=add 2024-09-16T08:12:21.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-16T08:12:21.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-79228-0-256 2024-09-16T08:12:21.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-79228-0-256 2024-09-16T08:12:21.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-16T08:12:21.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-16T08:12:21.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-79228-0-256 2024-09-16T08:12:21.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:12:21.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-16T08:12:21.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:12:21.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-16T08:12:21.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:12:21.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-16T08:12:21.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:12:21.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-16T08:12:21.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-79228-0-256 td/test-erasure-eio/ORIGINAL 2024-09-16T08:12:21.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-79228-0-256 td/test-erasure-eio 0 256 add 2024-09-16T08:12:21.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79228-0-256 2024-09-16T08:12:21.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-16T08:12:21.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-16T08:12:21.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-16T08:12:21.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=0 2024-09-16T08:12:21.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-16T08:12:21.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2024-09-16T08:12:21.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-16T08:12:21.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2024-09-16T08:12:21.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-16T08:12:21.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:12:21.537 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79228-0-256 2024-09-16T08:12:21.537 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:12:21.537 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79228-0-256 2024-09-16T08:12:21.537 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79228-0-256 2024-09-16T08:12:21.538 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:12:22.103 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:12:22.103 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:12:22.104 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T08:12:22.104 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-16T08:12:22.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-16T08:12:22.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=3 2024-09-16T08:12:22.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-16T08:12:23.043 INFO:tasks.workunit.client.0.smithi162.stderr:noout is set 2024-09-16T08:12:23.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2024-09-16T08:12:23.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-79228-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:12:23.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-16T08:12:23.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T08:12:23.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-16T08:12:23.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T08:12:23.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-79228-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:12:23.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-16T08:12:23.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T08:12:23.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-16T08:12:23.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T08:12:23.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-16T08:12:23.057 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:12:23.057 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:12:23.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:12:23.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:12:23.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:12:23.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:12:23.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-79228-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:12:23.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-16T08:12:23.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T08:12:23.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-16T08:12:23.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T08:12:23.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-09-16T08:12:23.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-79228-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:12:25.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-09-16T08:12:25.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-16T08:12:25.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T08:12:25.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-16T08:12:25.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T08:12:25.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-16T08:12:25.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:12:25.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:12:25.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:12:25.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:12:25.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-16T08:12:25.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-16T08:12:25.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T08:12:25.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T08:12:25.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:12:25.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T08:12:25.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:12:25.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:12:25.833 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:12:25.833 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:12:25.833 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:12:25.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:12:25.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:12:25.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:12:25.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:12:25.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:12:25.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:12:25.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:12:25.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:12:25.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:12:25.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T08:12:25.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T08:12:25.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-16T08:12:25.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-16T08:12:25.836 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.3 2024-09-16T08:12:25.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:12:25.837 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-16T08:12:25.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-16T08:12:25.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T08:12:25.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T08:12:25.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T08:12:25.872 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:12:25.867+0000 7f38ac5ac880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:12:25.877 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:12:25.871+0000 7f38ac5ac880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:12:25.879 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:12:25.875+0000 7f38ac5ac880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:12:26.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-16T08:12:26.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:12:26.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T08:12:26.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:12:26.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:12:26.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:12:26.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:12:26.342 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T08:12:26.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:12:26.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:12:26.424 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:12:26.419+0000 7f38ac5ac880 -1 Falling back to public interface 2024-09-16T08:12:26.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:12:27.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:12:27.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:12:27.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:12:27.820 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:12:27.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:12:27.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:12:28.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:12:29.152 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:12:29.147+0000 7f38ac5ac880 -1 osd.3 78 log_to_monitors true 2024-09-16T08:12:29.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:12:29.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:12:29.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:12:29.236 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:12:29.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:12:29.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:12:29.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:12:30.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:12:30.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:12:30.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:12:30.769 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T08:12:30.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:12:30.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:12:31.395 INFO:tasks.workunit.client.0.smithi162.stderr:osd.3 up in weight 1 up_from 84 up_thru 84 down_at 81 last_clean_interval [63,78) [v2:127.0.0.1:6826/3961767906,v1:127.0.0.1:6827/3961767906] [v2:127.0.0.1:6828/3961767906,v1:127.0.0.1:6829/3961767906] exists,up 44b0a228-3f30-4ac6-ba25-4769fa3c842f 2024-09-16T08:12:31.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:12:31.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:12:31.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:12:31.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-16T08:12:31.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:12:31.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:12:31.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:12:31.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:12:31.396 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:12:31.397 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:12:31.397 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:12:31.398 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:12:31.398 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:12:31.398 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:12:31.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:12:31.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:12:31.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:12:31.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:12:31.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:12:32.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:12:32.050 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:12:32.050 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:12:32.050 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:12:32.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:12:32.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:12:32.050 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:12:32.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803841 2024-09-16T08:12:32.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803841 2024-09-16T08:12:32.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803841' 2024-09-16T08:12:32.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:12:32.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:12:32.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=317827579912 2024-09-16T08:12:32.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 317827579912 2024-09-16T08:12:32.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803841 1-317827579912' 2024-09-16T08:12:32.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:12:32.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:12:32.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=77309411379 2024-09-16T08:12:32.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 77309411379 2024-09-16T08:12:32.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803841 1-317827579912 2-77309411379' 2024-09-16T08:12:32.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:12:32.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:12:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=360777252867 2024-09-16T08:12:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 360777252867 2024-09-16T08:12:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803841 1-317827579912 2-77309411379 3-360777252867' 2024-09-16T08:12:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:12:32.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803841 2024-09-16T08:12:32.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:12:32.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:12:32.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803841 2024-09-16T08:12:32.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:12:32.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803841 2024-09-16T08:12:32.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803841' 2024-09-16T08:12:32.770 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803841 2024-09-16T08:12:32.771 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:12:33.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803839 -lt 25769803841 2024-09-16T08:12:33.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T08:12:34.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T08:12:34.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:12:34.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803841 -lt 25769803841 2024-09-16T08:12:34.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:12:34.742 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-317827579912 2024-09-16T08:12:34.742 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:12:34.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:12:34.745 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-317827579912 2024-09-16T08:12:34.745 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:12:34.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=317827579912 2024-09-16T08:12:34.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 317827579912' 2024-09-16T08:12:34.747 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 317827579912 2024-09-16T08:12:34.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:12:35.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 317827579913 -lt 317827579912 2024-09-16T08:12:35.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:12:35.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:12:35.166 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-77309411379 2024-09-16T08:12:35.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:12:35.168 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-77309411379 2024-09-16T08:12:35.168 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:12:35.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=77309411379 2024-09-16T08:12:35.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 77309411379' 2024-09-16T08:12:35.171 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 77309411379 2024-09-16T08:12:35.171 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:12:35.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 77309411380 -lt 77309411379 2024-09-16T08:12:35.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:12:35.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-360777252867 2024-09-16T08:12:35.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:12:35.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:12:35.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-360777252867 2024-09-16T08:12:35.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:12:35.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=360777252867 2024-09-16T08:12:35.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 360777252867' 2024-09-16T08:12:35.579 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 360777252867 2024-09-16T08:12:35.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:12:36.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 360777252868 -lt 360777252867 2024-09-16T08:12:36.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:12:36.053 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:12:36.053 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:12:36.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:12:36.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:12:36.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:12:36.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:12:36.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:12:36.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:12:36.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:12:36.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:12:36.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:12:36.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:12:36.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:12:36.981 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:12:37.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:12:37.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:12:37.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:12:37.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2024-09-16T08:12:37.545 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records in 2024-09-16T08:12:37.545 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records out 2024-09-16T08:12:37.545 INFO:tasks.workunit.client.0.smithi162.stderr:256 bytes copied, 7.4705e-05 s, 3.4 MB/s 2024-09-16T08:12:37.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-79228-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:12:37.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-16T08:12:37.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T08:12:37.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-16T08:12:37.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T08:12:37.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-79228-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:12:37.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-16T08:12:37.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T08:12:37.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-16T08:12:37.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T08:12:37.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-16T08:12:37.546 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:12:37.546 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:12:37.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:12:37.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:12:37.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:12:37.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:12:37.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-79228-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:12:37.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-16T08:12:37.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T08:12:37.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-16T08:12:37.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T08:12:37.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-09-16T08:12:37.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-79228-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:12:40.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-09-16T08:12:40.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-16T08:12:40.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T08:12:40.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-16T08:12:40.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T08:12:40.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-16T08:12:40.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:12:40.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:12:40.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:12:40.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:12:40.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-16T08:12:40.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-16T08:12:40.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T08:12:40.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T08:12:40.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:12:40.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T08:12:40.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:12:40.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:12:40.250 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:12:40.251 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:12:40.251 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:12:40.251 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:12:40.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:12:40.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:12:40.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:12:40.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:12:40.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:12:40.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:12:40.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:12:40.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:12:40.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T08:12:40.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T08:12:40.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-16T08:12:40.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-16T08:12:40.252 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.3 2024-09-16T08:12:40.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:12:40.254 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-16T08:12:40.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-16T08:12:40.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T08:12:40.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T08:12:40.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T08:12:40.289 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:12:40.282+0000 7f9b53454880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:12:40.292 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:12:40.286+0000 7f9b53454880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:12:40.293 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:12:40.290+0000 7f9b53454880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:12:40.595 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:12:40.590+0000 7f9b53454880 -1 Falling back to public interface 2024-09-16T08:12:40.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-16T08:12:40.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:12:40.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T08:12:40.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:12:40.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:12:40.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:12:40.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:12:40.737 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T08:12:40.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:12:40.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:12:41.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:12:42.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:12:42.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:12:42.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:12:42.250 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:12:42.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:12:42.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:12:42.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:12:43.419 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:12:43.414+0000 7f9b53454880 -1 osd.3 86 log_to_monitors true 2024-09-16T08:12:43.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:12:43.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:12:43.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:12:43.751 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:12:43.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:12:43.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:12:44.469 INFO:tasks.workunit.client.0.smithi162.stderr:osd.3 up in weight 1 up_from 91 up_thru 84 down_at 87 last_clean_interval [84,86) [v2:127.0.0.1:6826/2515661014,v1:127.0.0.1:6827/2515661014] [v2:127.0.0.1:6828/2515661014,v1:127.0.0.1:6829/2515661014] exists,up 44b0a228-3f30-4ac6-ba25-4769fa3c842f 2024-09-16T08:12:44.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:12:44.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:12:44.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:12:44.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-16T08:12:44.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:12:44.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:12:44.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:12:44.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:12:44.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:12:44.472 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:12:44.472 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:12:44.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:12:44.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:12:44.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:12:44.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:12:44.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:12:44.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:12:44.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:12:44.678 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:12:45.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:12:45.353 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:12:45.353 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:12:45.353 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:12:45.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:12:45.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:12:45.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:12:45.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803846 2024-09-16T08:12:45.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803846 2024-09-16T08:12:45.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803846' 2024-09-16T08:12:45.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:12:45.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:12:45.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=317827579917 2024-09-16T08:12:45.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 317827579917 2024-09-16T08:12:45.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803846 1-317827579917' 2024-09-16T08:12:45.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:12:45.781 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:12:45.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=77309411384 2024-09-16T08:12:45.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 77309411384 2024-09-16T08:12:45.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803846 1-317827579917 2-77309411384' 2024-09-16T08:12:45.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:12:45.942 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:12:46.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=390842023939 2024-09-16T08:12:46.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 390842023939 2024-09-16T08:12:46.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803846 1-317827579917 2-77309411384 3-390842023939' 2024-09-16T08:12:46.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:12:46.104 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803846 2024-09-16T08:12:46.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:12:46.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:12:46.108 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803846 2024-09-16T08:12:46.108 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:12:46.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803846 2024-09-16T08:12:46.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803846' 2024-09-16T08:12:46.110 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803846 2024-09-16T08:12:46.110 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:12:46.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803846 -lt 25769803846 2024-09-16T08:12:46.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:12:46.575 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-317827579917 2024-09-16T08:12:46.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:12:46.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:12:46.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-317827579917 2024-09-16T08:12:46.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:12:46.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=317827579917 2024-09-16T08:12:46.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 317827579917' 2024-09-16T08:12:46.581 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 317827579917 2024-09-16T08:12:46.582 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:12:47.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 317827579917 -lt 317827579917 2024-09-16T08:12:47.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:12:47.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-77309411384 2024-09-16T08:12:47.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:12:47.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:12:47.043 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-77309411384 2024-09-16T08:12:47.043 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:12:47.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=77309411384 2024-09-16T08:12:47.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 77309411384' 2024-09-16T08:12:47.044 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 77309411384 2024-09-16T08:12:47.045 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:12:47.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 77309411382 -lt 77309411384 2024-09-16T08:12:47.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T08:12:48.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T08:12:48.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:12:48.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 77309411384 -lt 77309411384 2024-09-16T08:12:48.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:12:48.900 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-390842023939 2024-09-16T08:12:48.900 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:12:48.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:12:48.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-390842023939 2024-09-16T08:12:48.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:12:48.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=390842023939 2024-09-16T08:12:48.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 390842023939' 2024-09-16T08:12:48.905 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 390842023939 2024-09-16T08:12:48.906 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:12:49.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 390842023939 -lt 390842023939 2024-09-16T08:12:49.386 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:12:49.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:12:49.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:12:49.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:12:49.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:12:49.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:12:49.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:12:49.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:12:49.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:12:49.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:12:49.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:12:50.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:12:50.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:12:50.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:12:50.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:12:50.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:12:50.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:12:50.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:12:50.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-16T08:12:50.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-16T08:12:51.601 INFO:tasks.workunit.client.0.smithi162.stderr:noout is unset 2024-09-16T08:12:51.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79228-0-256 2024-09-16T08:12:51.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T08:12:51.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T08:12:51.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79228-0-256 2024-09-16T08:12:51.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-16T08:12:51.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-16T08:12:51.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-79228-0-256 td/test-erasure-eio/COPY 2024-09-16T08:12:51.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-16T08:12:51.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-16T08:12:51.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 0 + 1 2024-09-16T08:12:51.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=1 2024-09-16T08:12:51.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-79228-0-256 td/test-erasure-eio 1 256 add 2024-09-16T08:12:51.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79228-0-256 2024-09-16T08:12:51.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-16T08:12:51.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-16T08:12:51.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-16T08:12:51.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-09-16T08:12:51.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-16T08:12:51.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2024-09-16T08:12:51.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-16T08:12:51.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2024-09-16T08:12:51.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-16T08:12:51.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:12:51.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79228-0-256 2024-09-16T08:12:51.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:12:51.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79228-0-256 2024-09-16T08:12:51.791 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79228-0-256 2024-09-16T08:12:51.791 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:12:52.321 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:12:52.321 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:12:52.322 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T08:12:52.322 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-16T08:12:52.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-16T08:12:52.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-09-16T08:12:52.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-16T08:12:53.100 INFO:tasks.workunit.client.0.smithi162.stderr:noout is set 2024-09-16T08:12:53.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2024-09-16T08:12:53.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-79228-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:12:53.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-16T08:12:53.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T08:12:53.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-16T08:12:53.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T08:12:53.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-79228-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:12:53.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-16T08:12:53.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T08:12:53.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-16T08:12:53.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T08:12:53.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-16T08:12:53.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:12:53.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:12:53.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:12:53.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:12:53.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:12:53.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:12:53.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-79228-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:12:53.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-16T08:12:53.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T08:12:53.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-16T08:12:53.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T08:12:53.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-16T08:12:53.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-79228-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:12:55.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-16T08:12:55.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-16T08:12:55.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T08:12:55.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T08:12:55.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T08:12:55.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-16T08:12:55.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:12:55.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:12:55.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:12:55.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:12:55.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-16T08:12:55.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-16T08:12:55.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T08:12:55.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T08:12:55.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:12:55.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T08:12:55.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:12:55.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:12:55.801 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:12:55.801 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:12:55.802 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:12:55.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:12:55.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:12:55.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:12:55.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:12:55.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:12:55.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:12:55.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:12:55.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:12:55.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:12:55.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T08:12:55.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T08:12:55.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-16T08:12:55.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T08:12:55.805 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-09-16T08:12:55.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:12:55.806 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-16T08:12:55.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T08:12:55.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T08:12:55.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T08:12:55.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T08:12:55.844 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:12:55.842+0000 7fcf8aad7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:12:55.847 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:12:55.846+0000 7fcf8aad7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:12:55.853 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:12:55.850+0000 7fcf8aad7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:12:56.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T08:12:56.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:12:56.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T08:12:56.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:12:56.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:12:56.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:12:56.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:12:56.286 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T08:12:56.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:12:56.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:12:56.399 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:12:56.398+0000 7fcf8aad7880 -1 Falling back to public interface 2024-09-16T08:12:56.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:12:57.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:12:57.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:12:57.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:12:57.807 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:12:57.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:12:57.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:12:58.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:12:59.012 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:12:59.010+0000 7fcf8aad7880 -1 osd.1 95 log_to_monitors true 2024-09-16T08:12:59.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:12:59.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:12:59.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:12:59.232 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:12:59.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:12:59.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:12:59.886 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 101 up_thru 87 down_at 97 last_clean_interval [74,95) [v2:127.0.0.1:6810/2506049890,v1:127.0.0.1:6811/2506049890] [v2:127.0.0.1:6812/2506049890,v1:127.0.0.1:6813/2506049890] exists,up 2e5de68a-2e90-47df-b6bf-074eba0f06d7 2024-09-16T08:12:59.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:12:59.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:12:59.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:12:59.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-16T08:12:59.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:12:59.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:12:59.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:12:59.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:12:59.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:12:59.888 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:12:59.888 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:12:59.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:12:59.890 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:12:59.890 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:13:00.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:13:00.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:13:00.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:13:00.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:13:00.102 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:13:00.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:13:00.576 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:13:00.576 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:13:00.576 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:13:00.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:13:00.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:13:00.577 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:13:00.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803851 2024-09-16T08:13:00.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803851 2024-09-16T08:13:00.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803851' 2024-09-16T08:13:00.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:13:00.734 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:13:00.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=433791696899 2024-09-16T08:13:00.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 433791696899 2024-09-16T08:13:00.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803851 1-433791696899' 2024-09-16T08:13:00.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:13:00.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:13:01.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=77309411389 2024-09-16T08:13:01.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 77309411389 2024-09-16T08:13:01.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803851 1-433791696899 2-77309411389' 2024-09-16T08:13:01.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:13:01.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:13:01.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=390842023944 2024-09-16T08:13:01.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 390842023944 2024-09-16T08:13:01.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803851 1-433791696899 2-77309411389 3-390842023944' 2024-09-16T08:13:01.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:13:01.341 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803851 2024-09-16T08:13:01.341 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:13:01.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:13:01.344 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803851 2024-09-16T08:13:01.344 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:13:01.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803851 2024-09-16T08:13:01.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803851' 2024-09-16T08:13:01.347 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803851 2024-09-16T08:13:01.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:13:01.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803848 -lt 25769803851 2024-09-16T08:13:01.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T08:13:02.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T08:13:02.776 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:13:03.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803851 -lt 25769803851 2024-09-16T08:13:03.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:13:03.202 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-433791696899 2024-09-16T08:13:03.202 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:13:03.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:13:03.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-433791696899 2024-09-16T08:13:03.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:13:03.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=433791696899 2024-09-16T08:13:03.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 433791696899' 2024-09-16T08:13:03.208 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 433791696899 2024-09-16T08:13:03.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:13:03.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 433791696899 -lt 433791696899 2024-09-16T08:13:03.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:13:03.629 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-77309411389 2024-09-16T08:13:03.629 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:13:03.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:13:03.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-77309411389 2024-09-16T08:13:03.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:13:03.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=77309411389 2024-09-16T08:13:03.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 77309411389' 2024-09-16T08:13:03.635 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 77309411389 2024-09-16T08:13:03.635 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:13:04.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 77309411390 -lt 77309411389 2024-09-16T08:13:04.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:13:04.058 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-390842023944 2024-09-16T08:13:04.058 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:13:04.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:13:04.062 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-390842023944 2024-09-16T08:13:04.062 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:13:04.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=390842023944 2024-09-16T08:13:04.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 390842023944' 2024-09-16T08:13:04.064 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 390842023944 2024-09-16T08:13:04.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:13:04.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 390842023945 -lt 390842023944 2024-09-16T08:13:04.514 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:13:04.515 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:13:04.515 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:13:05.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:13:05.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:13:05.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:13:05.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:13:05.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:13:05.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:13:05.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:13:05.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:13:05.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T08:13:05.446 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:13:05.446 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:13:05.446 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:13:06.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-09-16T08:13:06.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-09-16T08:13:06.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T08:13:06.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-09-16T08:13:06.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T08:13:06.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T08:13:06.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T08:13:06.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:13:06.130 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:13:06.130 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:13:06.130 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:13:06.130 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:13:06.131 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:13:06.131 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:13:06.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:13:06.516 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:13:06.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:13:06.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:13:07.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:13:07.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:13:07.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:13:07.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2024-09-16T08:13:07.037 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records in 2024-09-16T08:13:07.038 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records out 2024-09-16T08:13:07.038 INFO:tasks.workunit.client.0.smithi162.stderr:256 bytes copied, 9.6705e-05 s, 2.6 MB/s 2024-09-16T08:13:07.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-79228-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:13:07.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-16T08:13:07.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T08:13:07.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-16T08:13:07.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T08:13:07.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-79228-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:13:07.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-16T08:13:07.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T08:13:07.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-16T08:13:07.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T08:13:07.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-16T08:13:07.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:13:07.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:13:07.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:13:07.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:13:07.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:13:07.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:13:07.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-79228-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:13:07.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-16T08:13:07.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T08:13:07.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-16T08:13:07.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T08:13:07.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-16T08:13:07.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-79228-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:13:09.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-16T08:13:09.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-16T08:13:09.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T08:13:09.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T08:13:09.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T08:13:09.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-16T08:13:09.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:13:09.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:13:09.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:13:09.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:13:09.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-16T08:13:09.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-16T08:13:09.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T08:13:09.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T08:13:09.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:13:09.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T08:13:09.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:13:09.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:13:09.846 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:13:09.846 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:13:09.847 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:13:09.848 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:13:09.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:13:09.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:13:09.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:13:09.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:13:09.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:13:09.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:13:09.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:13:09.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:13:09.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T08:13:09.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T08:13:09.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-16T08:13:09.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T08:13:09.849 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-09-16T08:13:09.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:13:09.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-16T08:13:09.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T08:13:09.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T08:13:09.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T08:13:09.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T08:13:09.887 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:13:09.886+0000 7f0597958880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:13:09.889 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:13:09.890+0000 7f0597958880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:13:09.892 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:13:09.890+0000 7f0597958880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:13:10.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T08:13:10.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:13:10.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T08:13:10.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:13:10.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:13:10.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:13:10.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:13:10.399 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T08:13:10.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:13:10.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:13:10.455 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:13:10.454+0000 7f0597958880 -1 Falling back to public interface 2024-09-16T08:13:10.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:13:11.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:13:11.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:13:11.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:13:11.827 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:13:11.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:13:11.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:13:12.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:13:12.830 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:13:12.830+0000 7f0597958880 -1 osd.1 104 log_to_monitors true 2024-09-16T08:13:13.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:13:13.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:13:13.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:13:13.266 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:13:13.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:13:13.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:13:13.868 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 108 up_thru 108 down_at 104 last_clean_interval [101,104) [v2:127.0.0.1:6810/2133162435,v1:127.0.0.1:6811/2133162435] [v2:127.0.0.1:6812/2133162435,v1:127.0.0.1:6813/2133162435] exists,up 2e5de68a-2e90-47df-b6bf-074eba0f06d7 2024-09-16T08:13:13.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:13:13.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:13:13.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:13:13.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-16T08:13:13.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:13:13.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:13:13.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:13:13.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:13:13.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:13:13.870 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:13:13.870 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:13:13.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:13:13.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:13:13.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:13:14.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:13:14.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:13:14.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:13:14.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:13:14.094 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:13:14.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:13:14.668 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:13:14.668 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:13:14.668 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:13:14.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:13:14.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:13:14.668 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:13:14.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803855 2024-09-16T08:13:14.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803855 2024-09-16T08:13:14.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803855' 2024-09-16T08:13:14.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:13:14.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:13:15.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=463856467971 2024-09-16T08:13:15.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 463856467971 2024-09-16T08:13:15.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803855 1-463856467971' 2024-09-16T08:13:15.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:13:15.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:13:15.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=77309411394 2024-09-16T08:13:15.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 77309411394 2024-09-16T08:13:15.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803855 1-463856467971 2-77309411394' 2024-09-16T08:13:15.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:13:15.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:13:15.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=390842023949 2024-09-16T08:13:15.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 390842023949 2024-09-16T08:13:15.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803855 1-463856467971 2-77309411394 3-390842023949' 2024-09-16T08:13:15.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:13:15.315 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803855 2024-09-16T08:13:15.315 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:13:15.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:13:15.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803855 2024-09-16T08:13:15.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:13:15.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803855 2024-09-16T08:13:15.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803855' 2024-09-16T08:13:15.321 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803855 2024-09-16T08:13:15.321 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:13:15.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803853 -lt 25769803855 2024-09-16T08:13:15.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T08:13:16.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T08:13:16.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:13:17.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803856 -lt 25769803855 2024-09-16T08:13:17.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:13:17.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-463856467971 2024-09-16T08:13:17.240 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:13:17.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:13:17.243 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-463856467971 2024-09-16T08:13:17.243 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:13:17.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=463856467971 2024-09-16T08:13:17.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 463856467971' 2024-09-16T08:13:17.245 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 463856467971 2024-09-16T08:13:17.246 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:13:17.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 463856467971 -lt 463856467971 2024-09-16T08:13:17.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:13:17.703 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-77309411394 2024-09-16T08:13:17.703 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:13:17.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:13:17.707 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-77309411394 2024-09-16T08:13:17.707 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:13:17.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=77309411394 2024-09-16T08:13:17.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 77309411394' 2024-09-16T08:13:17.709 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 77309411394 2024-09-16T08:13:17.710 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:13:18.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 77309411394 -lt 77309411394 2024-09-16T08:13:18.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:13:18.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-390842023949 2024-09-16T08:13:18.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:13:18.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:13:18.146 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-390842023949 2024-09-16T08:13:18.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:13:18.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=390842023949 2024-09-16T08:13:18.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 390842023949' 2024-09-16T08:13:18.149 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 390842023949 2024-09-16T08:13:18.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:13:18.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 390842023949 -lt 390842023949 2024-09-16T08:13:18.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:13:18.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:13:18.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:13:19.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:13:19.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:13:19.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:13:19.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:13:19.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:13:19.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:13:19.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:13:19.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:13:19.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:13:19.575 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:13:19.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:13:19.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:13:20.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:13:20.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:13:20.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:13:20.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-16T08:13:20.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-16T08:13:21.015 INFO:tasks.workunit.client.0.smithi162.stderr:noout is unset 2024-09-16T08:13:21.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79228-0-256 fail 2024-09-16T08:13:21.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T08:13:21.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T08:13:21.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79228-0-256 2024-09-16T08:13:21.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-16T08:13:21.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-16T08:13:21.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-79228-0-256 td/test-erasure-eio/COPY 2024-09-16T08:13:21.167 INFO:tasks.workunit.client.0.smithi162.stderr:error getting pool-jerasure/obj-size-79228-0-256: (5) Input/output error 2024-09-16T08:13:21.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-16T08:13:21.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-16T08:13:21.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:312: TEST_rados_get_bad_size_shard_0: delete_erasure_coded_pool pool-jerasure 2024-09-16T08:13:21.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T08:13:21.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-16T08:13:21.753 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-16T08:13:21.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-16T08:13:22.496 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-16T08:13:22.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-16T08:13:22.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T08:13:22.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T08:13:22.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T08:13:22.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:13:22.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:13:22.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:13:22.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:13:22.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:13:22.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:13:22.827 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T08:13:22.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T08:13:22.830 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T08:13:22.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T08:13:22.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T08:13:22.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T08:13:22.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:13:22.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T08:13:22.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T08:13:22.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:13:22.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:13:22.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T08:13:22.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T08:13:22.873 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T08:13:22.873 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:13:22.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:13:22.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T08:13:22.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T08:13:22.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T08:13:22.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-16T08:13:22.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-16T08:13:22.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-16T08:13:22.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-16T08:13:22.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T08:13:22.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T08:13:22.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T08:13:22.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:13:22.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:13:22.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:13:22.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:13:22.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:13:22.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:13:22.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T08:13:22.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T08:13:22.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T08:13:22.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T08:13:22.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T08:13:22.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T08:13:22.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:13:22.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T08:13:22.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T08:13:22.890 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:13:22.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:13:22.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T08:13:22.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T08:13:22.897 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T08:13:22.897 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:13:22.897 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:13:22.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T08:13:22.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T08:13:22.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T08:13:22.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-16T08:13:22.902 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T08:13:22.902 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:13:22.902 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:13:22.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79228 2024-09-16T08:13:22.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T08:13:22.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T08:13:22.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T08:13:22.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-16T08:13:22.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-16T08:13:22.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-16T08:13:22.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T08:13:22.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T08:13:22.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T08:13:22.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-16T08:13:22.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-16T08:13:23.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T08:13:23.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:13:23.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:13:23.376 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:13:23.376 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:13:23.376 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:13:23.376 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:13:23.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T08:13:23.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T08:13:23.623 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T08:13:23.623 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T08:13:23.623 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T08:13:23.623 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T08:13:23.624 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T08:13:23.624 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T08:13:23.625 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T08:13:23.625 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T08:13:23.625 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:13:23.626 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:13:23.626 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:13:23.626 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T08:13:23.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T08:13:23.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get fsid 2024-09-16T08:13:23.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T08:13:23.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T08:13:23.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T08:13:23.726 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T08:13:23.726 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T08:13:23.726 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T08:13:23.726 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T08:13:23.727 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T08:13:23.727 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:13:23.727 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:13:23.728 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:13:23.728 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T08:13:23.729 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T08:13:23.729 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get mon_host 2024-09-16T08:13:23.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-16T08:13:23.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-16T08:13:23.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T08:13:23.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T08:13:23.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T08:13:23.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-16T08:13:23.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T08:13:24.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T08:13:24.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:13:24.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:13:24.740 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:13:24.740 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:13:24.740 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:13:24.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:13:24.742 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T08:13:24.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T08:13:24.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-16T08:13:24.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T08:13:25.219 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-16T08:13:25.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T08:13:26.231 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-16T08:13:26.231 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T08:13:26.231 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T08:13:26.232 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:13:26.232 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:13:26.232 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:13:26.232 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T08:13:26.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-16T08:13:26.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-mon.a.asok log flush 2024-09-16T08:13:26.316 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T08:13:26.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-16T08:13:26.327 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T08:13:23.433+0000 7fdfe56c5d40 0 load: jerasure load: lrc 2024-09-16T08:13:26.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_1 td/test-erasure-eio 2024-09-16T08:13:26.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:316: TEST_rados_get_bad_size_shard_1: local dir=td/test-erasure-eio 2024-09-16T08:13:26.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:317: TEST_rados_get_bad_size_shard_1: setup_osds 4 2024-09-16T08:13:26.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-16T08:13:26.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-16T08:13:26.329 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-16T08:13:26.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-16T08:13:26.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:13:26.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-16T08:13:26.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:13:26.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:13:26.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T08:13:26.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:13:26.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-16T08:13:26.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:13:26.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:13:26.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:13:26.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:13:26.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-16T08:13:26.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-16T08:13:26.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:13:26.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:13:26.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:13:26.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:13:26.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:13:26.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:13:26.334 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:13:26.334 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:13:26.335 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:13:26.337 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:13:26.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:13:26.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:13:26.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:13:26.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:13:26.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:13:26.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:13:26.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:13:26.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:13:26.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:13:26.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:13:26.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:13:26.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:13:26.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-16T08:13:26.338 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:13:26.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=858d12c6-f467-48db-b584-a1e42461049c 2024-09-16T08:13:26.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 858d12c6-f467-48db-b584-a1e42461049c' 2024-09-16T08:13:26.339 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 858d12c6-f467-48db-b584-a1e42461049c 2024-09-16T08:13:26.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:13:26.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCm6OdmkhmyFRAApOhkBL5ClV05NsKnA4KERg== 2024-09-16T08:13:26.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCm6OdmkhmyFRAApOhkBL5ClV05NsKnA4KERg=="}' 2024-09-16T08:13:26.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 858d12c6-f467-48db-b584-a1e42461049c -i td/test-erasure-eio/0/new.json 2024-09-16T08:13:26.727 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:13:26.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-16T08:13:26.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCm6OdmkhmyFRAApOhkBL5ClV05NsKnA4KERg== --osd-uuid 858d12c6-f467-48db-b584-a1e42461049c 2024-09-16T08:13:26.778 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:13:26.777+0000 7fe148b2d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:13:26.782 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:13:26.781+0000 7fe148b2d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:13:26.785 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:13:26.781+0000 7fe148b2d880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:13:26.786 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:13:26.785+0000 7fe148b2d880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T08:13:26.786 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:13:26.785+0000 7fe148b2d880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T08:13:26.786 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:13:26.785+0000 7fe148b2d880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T08:13:26.897 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:13:26.893+0000 7fe148b2d880 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-16T08:13:32.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-16T08:13:32.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:13:32.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T08:13:32.597 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T08:13:32.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:13:33.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T08:13:33.174 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T08:13:33.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:13:33.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:13:33.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:13:33.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:13:33.207 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:13:33.205+0000 7feb835de880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:13:33.210 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:13:33.209+0000 7feb835de880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:13:33.212 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:13:33.209+0000 7feb835de880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:13:33.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T08:13:33.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:13:33.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T08:13:33.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:13:33.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:13:33.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:13:33.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:13:33.614 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:13:33.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:13:33.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:13:33.763 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:13:33.761+0000 7feb835de880 -1 Falling back to public interface 2024-09-16T08:13:34.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:13:35.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:13:35.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:13:35.080 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:13:35.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:13:35.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:13:35.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:13:35.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:13:36.359 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:13:36.357+0000 7feb835de880 -1 osd.0 0 log_to_monitors true 2024-09-16T08:13:36.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:13:36.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:13:36.492 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:13:36.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:13:36.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:13:36.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:13:37.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:13:38.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:13:38.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:13:38.097 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:13:38.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:13:38.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:13:38.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:13:38.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:13:39.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:13:39.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:13:39.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:13:39.576 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:13:39.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:13:39.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:13:40.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:13:41.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:13:41.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:13:41.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:13:41.102 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:13:41.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:13:41.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:13:41.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:13:42.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:13:42.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:13:42.560 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:13:42.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:13:42.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:13:42.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:13:43.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:13:44.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:13:44.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:13:44.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:13:44.094 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:13:44.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:13:44.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:13:44.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:13:45.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:13:45.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:13:45.564 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:13:45.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:13:45.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:13:45.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:13:46.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:13:47.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:13:47.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:13:47.096 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:13:47.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:13:47.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:13:47.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:13:47.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:13:48.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:13:48.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:13:48.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:13:48.569 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:13:48.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:13:48.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:13:49.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:13:50.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:13:50.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:13:50.159 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:13:50.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:13:50.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:13:50.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:13:51.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:13:52.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:13:52.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:13:52.254 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:13:52.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:13:52.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:13:52.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:13:53.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:13:54.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:13:54.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:13:54.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:13:54.041 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:13:54.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:13:54.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:13:54.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:13:55.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:13:55.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:13:55.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:13:55.899 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:13:55.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:13:55.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:13:56.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:13:57.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:13:57.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:13:57.632 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:13:57.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:13:57.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:13:57.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:13:58.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:13:59.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:13:59.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:13:59.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:13:59.953 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:13:59.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:13:59.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:14:00.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:01.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:01.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:01.799 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:14:01.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:14:01.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:01.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:14:02.417 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1113184129,v1:127.0.0.1:6803/1113184129] [v2:127.0.0.1:6804/1113184129,v1:127.0.0.1:6805/1113184129] exists,up 858d12c6-f467-48db-b584-a1e42461049c 2024-09-16T08:14:02.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:14:02.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:14:02.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:14:02.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:14:02.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-16T08:14:02.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:14:02.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:14:02.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T08:14:02.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:14:02.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-16T08:14:02.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:14:02.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:14:02.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:14:02.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:14:02.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-16T08:14:02.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-16T08:14:02.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:14:02.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:14:02.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:14:02.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:14:02.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:14:02.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:14:02.421 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:14:02.421 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:14:02.421 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:14:02.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:14:02.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:14:02.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:14:02.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:14:02.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:14:02.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:14:02.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:14:02.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:14:02.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:14:02.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:14:02.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:14:02.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:14:02.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:14:02.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-16T08:14:02.425 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:14:02.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=58db0e1e-7376-4e4a-85b3-d7201fbaf42d 2024-09-16T08:14:02.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 58db0e1e-7376-4e4a-85b3-d7201fbaf42d' 2024-09-16T08:14:02.426 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 58db0e1e-7376-4e4a-85b3-d7201fbaf42d 2024-09-16T08:14:02.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:14:02.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDK6Odmg1XdGhAA0k7/hNBWlZUkz+Zh4gmuxQ== 2024-09-16T08:14:02.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDK6Odmg1XdGhAA0k7/hNBWlZUkz+Zh4gmuxQ=="}' 2024-09-16T08:14:02.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 58db0e1e-7376-4e4a-85b3-d7201fbaf42d -i td/test-erasure-eio/1/new.json 2024-09-16T08:14:02.989 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:14:03.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-16T08:14:03.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDK6Odmg1XdGhAA0k7/hNBWlZUkz+Zh4gmuxQ== --osd-uuid 58db0e1e-7376-4e4a-85b3-d7201fbaf42d 2024-09-16T08:14:03.041 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:14:03.037+0000 7f5352aba880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:14:03.044 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:14:03.041+0000 7f5352aba880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:14:03.046 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:14:03.045+0000 7f5352aba880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:14:03.047 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:14:03.045+0000 7f5352aba880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T08:14:03.047 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:14:03.045+0000 7f5352aba880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T08:14:03.047 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:14:03.045+0000 7f5352aba880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T08:14:03.145 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:14:03.141+0000 7f5352aba880 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-16T08:14:09.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-16T08:14:09.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:14:09.740 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-16T08:14:09.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T08:14:09.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:14:10.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T08:14:10.321 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-16T08:14:10.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:14:10.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:14:10.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:14:10.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:14:10.347 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:14:10.344+0000 7f9ddbf17880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:14:10.350 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:14:10.348+0000 7f9ddbf17880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:14:10.351 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:14:10.348+0000 7f9ddbf17880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:14:10.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T08:14:10.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:14:10.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T08:14:10.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:14:10.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:14:10.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:10.757 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:14:10.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:14:10.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:10.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:14:10.920 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:14:10.916+0000 7f9ddbf17880 -1 Falling back to public interface 2024-09-16T08:14:11.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:12.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:12.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:12.204 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:14:12.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:14:12.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:12.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:14:12.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:13.511 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:14:13.508+0000 7f9ddbf17880 -1 osd.1 0 log_to_monitors true 2024-09-16T08:14:13.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:13.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:13.657 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:14:13.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:14:13.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:13.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:14:14.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:15.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:15.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:15.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:14:15.217 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:14:15.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:15.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:14:15.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:16.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:16.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:16.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:14:16.738 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:14:16.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:16.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:14:17.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:18.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:18.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:18.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:14:18.250 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:14:18.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:18.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:14:18.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:19.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:19.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:19.768 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:14:19.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:14:19.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:19.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:14:20.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:21.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:21.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:21.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:14:21.314 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:14:21.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:21.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:14:21.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:22.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:22.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:22.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:14:22.798 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:14:22.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:22.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:14:23.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:24.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:24.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:24.242 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:14:24.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:14:24.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:24.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:14:24.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:25.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:25.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:25.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:14:25.771 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:14:25.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:25.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:14:26.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:27.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:27.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:27.340 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:14:27.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:14:27.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:27.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:14:28.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:29.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:29.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:29.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:14:29.541 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:14:29.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:29.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:14:30.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:31.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:31.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:31.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:14:31.135 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:14:31.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:31.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:14:31.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:32.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:32.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:32.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:14:32.609 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:14:32.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:32.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:14:33.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:14:34.255 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:14:34.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:34.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:14:34.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:35.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:35.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:35.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:14:35.929 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:14:35.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:35.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:14:36.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:37.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:37.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:37.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:14:37.678 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:14:37.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:37.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:14:38.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:39.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:39.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:39.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T08:14:39.580 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-16T08:14:39.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:39.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:14:39.981 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3819531503,v1:127.0.0.1:6811/3819531503] [v2:127.0.0.1:6812/3819531503,v1:127.0.0.1:6813/3819531503] exists,up 58db0e1e-7376-4e4a-85b3-d7201fbaf42d 2024-09-16T08:14:39.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:14:39.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:14:39.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:14:39.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:14:39.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-16T08:14:39.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:14:39.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:14:39.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T08:14:39.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:14:39.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-16T08:14:39.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:14:39.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:14:39.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:14:39.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:14:39.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-16T08:14:39.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-16T08:14:39.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:14:39.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:14:39.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:14:39.985 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:14:39.985 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:14:39.985 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:14:39.985 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:14:39.985 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:14:39.985 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:14:39.987 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:14:39.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:14:39.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:14:39.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:14:39.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:14:39.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:14:39.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:14:39.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:14:39.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:14:39.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:14:39.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:14:39.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:14:39.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:14:39.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-16T08:14:39.988 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:14:39.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1ea0584f-efa7-44ae-95cb-2a006c2ccdf8 2024-09-16T08:14:39.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 1ea0584f-efa7-44ae-95cb-2a006c2ccdf8' 2024-09-16T08:14:39.989 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 1ea0584f-efa7-44ae-95cb-2a006c2ccdf8 2024-09-16T08:14:39.989 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:14:40.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDw6Odm8yzhABAALOVheGIBbmt1i3FA+fCZrg== 2024-09-16T08:14:40.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDw6Odm8yzhABAALOVheGIBbmt1i3FA+fCZrg=="}' 2024-09-16T08:14:40.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1ea0584f-efa7-44ae-95cb-2a006c2ccdf8 -i td/test-erasure-eio/2/new.json 2024-09-16T08:14:40.656 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:14:40.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-16T08:14:40.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDw6Odm8yzhABAALOVheGIBbmt1i3FA+fCZrg== --osd-uuid 1ea0584f-efa7-44ae-95cb-2a006c2ccdf8 2024-09-16T08:14:40.712 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:14:40.708+0000 7f9036ce0880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:14:40.737 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:14:40.736+0000 7f9036ce0880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:14:40.739 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:14:40.736+0000 7f9036ce0880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:14:40.740 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:14:40.736+0000 7f9036ce0880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:14:40.740 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:14:40.736+0000 7f9036ce0880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:14:40.740 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:14:40.736+0000 7f9036ce0880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:14:40.834 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:14:40.832+0000 7f9036ce0880 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-16T08:14:47.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-16T08:14:47.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:14:47.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T08:14:47.281 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-16T08:14:47.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:14:47.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T08:14:47.838 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-16T08:14:47.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:14:47.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:14:47.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:14:47.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:14:47.872 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:14:47.868+0000 7fabb49b5880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:14:47.905 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:14:47.904+0000 7fabb49b5880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:14:47.907 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:14:47.904+0000 7fabb49b5880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:14:48.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T08:14:48.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:14:48.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T08:14:48.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:14:48.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:14:48.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:48.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:14:48.347 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:14:48.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:48.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:14:48.488 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:14:48.484+0000 7fabb49b5880 -1 Falling back to public interface 2024-09-16T08:14:48.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:49.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:49.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:49.779 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:14:49.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:14:49.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:49.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:14:50.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:51.103 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:14:51.100+0000 7fabb49b5880 -1 osd.2 0 log_to_monitors true 2024-09-16T08:14:51.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:51.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:51.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:14:51.255 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:14:51.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:51.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:14:51.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:52.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:52.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:52.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:14:52.841 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:14:52.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:52.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:14:53.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:54.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:54.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:54.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:14:54.393 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:14:54.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:54.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:14:54.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:55.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:55.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:55.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:14:55.819 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:14:55.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:55.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:14:56.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:57.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:57.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:57.258 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:14:57.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:14:57.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:57.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:14:57.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:14:58.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:14:58.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:14:58.725 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:14:58.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:14:58.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:14:58.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:14:59.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:00.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:00.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:00.285 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:15:00.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:15:00.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:00.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:15:00.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:01.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:01.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:01.850 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:15:01.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:15:01.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:01.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:15:02.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:03.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:03.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:03.280 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:15:03.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:15:03.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:03.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:15:03.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:04.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:04.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:04.712 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:15:04.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:15:04.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:04.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:15:05.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:06.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:06.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:06.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:15:06.739 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:15:06.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:06.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:15:07.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:08.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:08.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:08.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:15:08.437 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:15:08.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:08.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:15:09.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:10.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:10.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:10.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:15:10.360 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:15:10.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:10.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:15:11.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:12.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:12.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:12.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:15:12.375 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:15:12.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:12.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:15:13.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:14.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:14.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:14.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:15:14.338 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:15:14.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:14.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:15:15.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:16.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:16.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:16.162 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:15:16.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:15:16.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:16.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:15:17.005 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3425454285,v1:127.0.0.1:6819/3425454285] [v2:127.0.0.1:6820/3425454285,v1:127.0.0.1:6821/3425454285] exists,up 1ea0584f-efa7-44ae-95cb-2a006c2ccdf8 2024-09-16T08:15:17.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:15:17.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:15:17.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:15:17.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:15:17.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-16T08:15:17.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:15:17.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:15:17.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T08:15:17.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:15:17.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-16T08:15:17.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:15:17.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:15:17.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:15:17.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:15:17.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-16T08:15:17.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-16T08:15:17.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:15:17.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:15:17.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:15:17.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:15:17.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:15:17.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:15:17.009 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:15:17.009 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:15:17.010 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:15:17.010 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:15:17.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:15:17.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:15:17.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:15:17.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:15:17.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:15:17.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:15:17.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:15:17.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:15:17.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:15:17.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:15:17.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:15:17.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:15:17.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-16T08:15:17.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:15:17.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b2090afa-db06-4b8e-9842-a8a60d192e25 2024-09-16T08:15:17.013 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 b2090afa-db06-4b8e-9842-a8a60d192e25 2024-09-16T08:15:17.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 b2090afa-db06-4b8e-9842-a8a60d192e25' 2024-09-16T08:15:17.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:15:17.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAV6edmI9ZfAhAAniE8hwGHbCd1avlIjSO1Ww== 2024-09-16T08:15:17.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAV6edmI9ZfAhAAniE8hwGHbCd1avlIjSO1Ww=="}' 2024-09-16T08:15:17.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b2090afa-db06-4b8e-9842-a8a60d192e25 -i td/test-erasure-eio/3/new.json 2024-09-16T08:15:17.711 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:15:17.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-16T08:15:17.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAV6edmI9ZfAhAAniE8hwGHbCd1avlIjSO1Ww== --osd-uuid b2090afa-db06-4b8e-9842-a8a60d192e25 2024-09-16T08:15:17.762 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:15:17.759+0000 7f052d8d6880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:15:17.873 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:15:17.871+0000 7f052d8d6880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:15:17.876 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:15:17.871+0000 7f052d8d6880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:15:17.877 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:15:17.871+0000 7f052d8d6880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:15:17.877 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:15:17.871+0000 7f052d8d6880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:15:17.877 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:15:17.871+0000 7f052d8d6880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:15:17.923 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:15:17.919+0000 7f052d8d6880 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-16T08:15:24.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-16T08:15:24.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:15:24.437 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-16T08:15:24.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T08:15:24.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:15:24.998 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-16T08:15:24.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T08:15:24.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:15:25.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:15:25.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:15:25.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:15:25.032 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:15:25.027+0000 7f3a35dec880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:15:25.035 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:15:25.031+0000 7f3a35dec880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:15:25.036 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:15:25.031+0000 7f3a35dec880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:15:25.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T08:15:25.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:15:25.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T08:15:25.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:15:25.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:15:25.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:25.456 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:15:25.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:15:25.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:25.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:15:25.619 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:15:25.615+0000 7f3a35dec880 -1 Falling back to public interface 2024-09-16T08:15:25.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:26.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:26.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:26.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:15:26.876 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:15:26.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:26.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:15:27.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:28.053 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:15:28.047+0000 7f3a35dec880 -1 osd.3 0 log_to_monitors true 2024-09-16T08:15:28.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:28.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:28.363 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:15:28.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:15:28.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:28.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:15:29.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:30.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:30.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:30.125 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:15:30.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:15:30.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:30.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:15:30.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:31.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:31.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:31.724 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:15:31.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:15:31.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:31.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:15:32.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:33.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:33.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:33.246 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:15:33.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:15:33.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:33.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:15:33.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:34.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:34.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:34.757 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:15:34.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:15:34.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:34.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:15:35.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:36.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:36.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:36.195 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:15:36.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:15:36.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:36.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:15:36.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:37.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:37.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:37.689 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:15:37.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:15:37.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:37.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:15:38.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:39.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:39.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:39.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:15:39.234 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:15:39.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:39.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:15:39.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:40.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:40.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:40.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:15:40.683 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:15:40.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:40.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:15:41.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:42.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:42.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:42.246 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:15:42.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:15:42.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:42.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:15:42.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:43.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:43.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:43.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:15:43.813 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:15:43.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:43.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:15:44.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:45.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:45.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:45.427 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:15:45.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:15:45.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:45.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:15:46.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:47.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:47.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:47.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:15:47.354 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:15:47.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:47.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:15:47.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:48.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:48.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:48.821 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:15:48.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:15:48.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:48.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:15:49.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:50.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:50.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:50.374 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:15:50.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:15:50.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:50.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:15:51.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:52.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:52.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:52.410 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:15:52.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:15:52.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:52.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:15:53.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:15:54.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:15:54.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:15:54.387 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-16T08:15:54.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T08:15:54.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:15:54.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:15:54.992 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3663046769,v1:127.0.0.1:6827/3663046769] [v2:127.0.0.1:6828/3663046769,v1:127.0.0.1:6829/3663046769] exists,up b2090afa-db06-4b8e-9842-a8a60d192e25 2024-09-16T08:15:54.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:15:54.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:15:54.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:15:54.993 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-16T08:15:54.993 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T08:15:54.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T08:15:54.994 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:15:54.994 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:15:54.994 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:15:54.995 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.0.asok 2024-09-16T08:15:54.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-16T08:15:54.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.0.asok log flush 2024-09-16T08:15:55.091 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T08:15:55.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-16T08:15:55.106 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T08:13:33.773+0000 7feb835de880 0 load: jerasure load: lrc 2024-09-16T08:15:55.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:319: TEST_rados_get_bad_size_shard_1: local poolname=pool-jerasure 2024-09-16T08:15:55.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:320: TEST_rados_get_bad_size_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2024-09-16T08:15:55.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T08:15:55.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-16T08:15:55.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-16T08:15:55.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-16T08:15:55.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-16T08:15:55.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-16T08:15:55.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-16T08:15:56.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-16T08:15:56.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-16T08:15:57.558 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-16T08:15:57.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T08:15:58.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-16T08:15:58.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:15:58.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:15:58.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:15:58.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:15:58.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:15:58.571 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:15:58.571 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:15:58.572 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:15:58.572 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:15:58.572 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:15:58.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:15:58.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:15:58.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:15:58.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:15:58.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:15:59.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:15:59.305 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:15:59.305 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:15:59.305 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:15:59.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:15:59.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:15:59.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:15:59.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803812 2024-09-16T08:15:59.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803812 2024-09-16T08:15:59.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803812' 2024-09-16T08:15:59.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:15:59.658 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:15:59.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=55834574884 2024-09-16T08:15:59.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 55834574884 2024-09-16T08:15:59.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803812 1-55834574884' 2024-09-16T08:15:59.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:15:59.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:16:00.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378645 2024-09-16T08:16:00.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378645 2024-09-16T08:16:00.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803812 1-55834574884 2-81604378645' 2024-09-16T08:16:00.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:16:00.077 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:16:00.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=120259084306 2024-09-16T08:16:00.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 120259084306 2024-09-16T08:16:00.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803812 1-55834574884 2-81604378645 3-120259084306' 2024-09-16T08:16:00.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:16:00.232 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803812 2024-09-16T08:16:00.232 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:16:00.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:16:00.235 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803812 2024-09-16T08:16:00.235 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:16:00.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803812 2024-09-16T08:16:00.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803812' 2024-09-16T08:16:00.237 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803812 2024-09-16T08:16:00.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:16:00.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803812 -lt 25769803812 2024-09-16T08:16:00.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:16:00.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:16:00.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-55834574884 2024-09-16T08:16:00.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:16:00.665 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-55834574884 2024-09-16T08:16:00.665 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:16:00.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=55834574884 2024-09-16T08:16:00.668 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574884 2024-09-16T08:16:00.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 55834574884' 2024-09-16T08:16:00.668 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:16:01.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 55834574884 -lt 55834574884 2024-09-16T08:16:01.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:16:01.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378645 2024-09-16T08:16:01.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:16:01.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:16:01.202 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378645 2024-09-16T08:16:01.203 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:16:01.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378645 2024-09-16T08:16:01.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378645' 2024-09-16T08:16:01.205 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 81604378645 2024-09-16T08:16:01.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:16:01.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378645 -lt 81604378645 2024-09-16T08:16:01.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:16:01.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-120259084306 2024-09-16T08:16:01.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:16:01.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:16:01.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-120259084306 2024-09-16T08:16:01.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:16:01.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=120259084306 2024-09-16T08:16:01.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 120259084306' 2024-09-16T08:16:01.654 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 120259084306 2024-09-16T08:16:01.654 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:16:02.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 120259084306 -lt 120259084306 2024-09-16T08:16:02.149 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:16:02.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:16:02.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:16:02.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:16:02.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:16:02.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:16:02.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:16:02.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:16:02.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:16:02.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:16:02.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:16:03.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:16:03.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:16:03.182 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:16:03.182 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:16:03.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:16:03.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:16:03.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:16:03.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:322: TEST_rados_get_bad_size_shard_1: local shard_id=1 2024-09-16T08:16:03.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:323: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 10 2024-09-16T08:16:03.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-16T08:16:03.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-09-16T08:16:03.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=1 2024-09-16T08:16:03.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-09-16T08:16:03.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=10 2024-09-16T08:16:03.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-09-16T08:16:03.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-09-16T08:16:03.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-16T08:16:03.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-79228-1-10 2024-09-16T08:16:03.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-79228-1-10 2024-09-16T08:16:03.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-16T08:16:03.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-16T08:16:03.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-79228-1-10 2024-09-16T08:16:03.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:16:03.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-16T08:16:03.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:16:03.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-16T08:16:03.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:16:03.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-16T08:16:03.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:16:03.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-16T08:16:03.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-79228-1-10 td/test-erasure-eio/ORIGINAL 2024-09-16T08:16:03.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-79228-1-10 td/test-erasure-eio 1 10 set 2024-09-16T08:16:03.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79228-1-10 2024-09-16T08:16:03.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-16T08:16:03.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-16T08:16:03.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-16T08:16:03.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-09-16T08:16:03.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-16T08:16:03.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-09-16T08:16:03.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-16T08:16:03.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-09-16T08:16:03.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-16T08:16:03.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:16:03.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79228-1-10 2024-09-16T08:16:03.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:16:03.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79228-1-10 2024-09-16T08:16:03.883 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79228-1-10 2024-09-16T08:16:03.883 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:16:04.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:16:04.340 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:16:04.340 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T08:16:04.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-16T08:16:04.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-16T08:16:04.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-09-16T08:16:04.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-16T08:16:05.104 INFO:tasks.workunit.client.0.smithi162.stderr:noout is set 2024-09-16T08:16:05.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-09-16T08:16:05.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-09-16T08:16:05.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-09-16T08:16:05.121 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records in 2024-09-16T08:16:05.122 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records out 2024-09-16T08:16:05.122 INFO:tasks.workunit.client.0.smithi162.stderr:10 bytes copied, 0.000191319 s, 52.3 kB/s 2024-09-16T08:16:05.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-79228-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:16:05.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-16T08:16:05.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T08:16:05.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-16T08:16:05.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T08:16:05.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-79228-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:16:05.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-16T08:16:05.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T08:16:05.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-16T08:16:05.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T08:16:05.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-16T08:16:05.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:16:05.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:16:05.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:16:05.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:16:05.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:16:05.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:16:05.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-79228-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:16:05.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-16T08:16:05.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T08:16:05.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-16T08:16:05.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T08:16:05.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-16T08:16:05.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-79228-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:16:08.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-16T08:16:08.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-16T08:16:08.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T08:16:08.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T08:16:08.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T08:16:08.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-16T08:16:08.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:16:08.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:16:08.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:16:08.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:16:08.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-16T08:16:08.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-16T08:16:08.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T08:16:08.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T08:16:08.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:16:08.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T08:16:08.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:16:08.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:16:08.207 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:16:08.207 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:16:08.207 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:16:08.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:16:08.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:16:08.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:16:08.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:16:08.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:16:08.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:16:08.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:16:08.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:16:08.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:16:08.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T08:16:08.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T08:16:08.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-16T08:16:08.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T08:16:08.208 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-09-16T08:16:08.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:16:08.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-16T08:16:08.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T08:16:08.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T08:16:08.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T08:16:08.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T08:16:08.244 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:16:08.242+0000 7fb0d79b6880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:16:08.247 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:16:08.246+0000 7fb0d79b6880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:16:08.253 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:16:08.250+0000 7fb0d79b6880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:16:08.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T08:16:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:16:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T08:16:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:16:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:16:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:16:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:16:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T08:16:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:16:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:16:08.807 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:16:08.806+0000 7fb0d79b6880 -1 Falling back to public interface 2024-09-16T08:16:09.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:16:10.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:16:10.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:16:10.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:16:10.231 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:16:10.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:16:10.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:16:10.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:16:11.259 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:16:11.258+0000 7fb0d79b6880 -1 osd.1 36 log_to_monitors true 2024-09-16T08:16:11.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:16:11.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:16:11.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:16:11.671 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:16:11.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:16:11.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:16:12.373 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 42 up_thru 42 down_at 38 last_clean_interval [13,36) [v2:127.0.0.1:6810/4118494955,v1:127.0.0.1:6811/4118494955] [v2:127.0.0.1:6812/4118494955,v1:127.0.0.1:6813/4118494955] exists,up 58db0e1e-7376-4e4a-85b3-d7201fbaf42d 2024-09-16T08:16:12.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:16:12.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:16:12.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:16:12.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-16T08:16:12.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:16:12.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:16:12.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:16:12.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:16:12.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:16:12.376 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:16:12.376 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:16:12.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:16:12.378 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:16:12.378 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:16:12.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:16:12.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:16:12.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:16:12.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:16:12.589 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:16:13.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:16:13.188 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:16:13.188 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:16:13.189 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:16:13.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:16:13.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:16:13.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:16:13.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803817 2024-09-16T08:16:13.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803817 2024-09-16T08:16:13.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803817' 2024-09-16T08:16:13.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:16:13.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:16:13.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=180388626435 2024-09-16T08:16:13.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 180388626435 2024-09-16T08:16:13.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803817 1-180388626435' 2024-09-16T08:16:13.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:16:13.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:16:13.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378650 2024-09-16T08:16:13.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378650 2024-09-16T08:16:13.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803817 1-180388626435 2-81604378650' 2024-09-16T08:16:13.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:16:13.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:16:13.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=120259084311 2024-09-16T08:16:13.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 120259084311 2024-09-16T08:16:13.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803817 1-180388626435 2-81604378650 3-120259084311' 2024-09-16T08:16:13.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:16:13.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803817 2024-09-16T08:16:13.924 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:16:13.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:16:13.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803817 2024-09-16T08:16:13.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:16:13.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803817 2024-09-16T08:16:13.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803817' 2024-09-16T08:16:13.929 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803817 2024-09-16T08:16:13.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:16:14.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803815 -lt 25769803817 2024-09-16T08:16:14.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T08:16:15.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T08:16:15.361 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:16:15.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803817 -lt 25769803817 2024-09-16T08:16:15.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:16:15.808 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-180388626435 2024-09-16T08:16:15.808 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:16:15.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:16:15.810 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-180388626435 2024-09-16T08:16:15.810 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:16:15.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=180388626435 2024-09-16T08:16:15.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 180388626435' 2024-09-16T08:16:15.812 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 180388626435 2024-09-16T08:16:15.813 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:16:16.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 180388626435 -lt 180388626435 2024-09-16T08:16:16.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:16:16.234 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378650 2024-09-16T08:16:16.234 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:16:16.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:16:16.237 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378650 2024-09-16T08:16:16.237 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:16:16.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378650 2024-09-16T08:16:16.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378650' 2024-09-16T08:16:16.238 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 81604378650 2024-09-16T08:16:16.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:16:16.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378650 -lt 81604378650 2024-09-16T08:16:16.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:16:16.710 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:16:16.710 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-120259084311 2024-09-16T08:16:16.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:16:16.713 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-120259084311 2024-09-16T08:16:16.713 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:16:16.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=120259084311 2024-09-16T08:16:16.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 120259084311' 2024-09-16T08:16:16.715 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 120259084311 2024-09-16T08:16:16.716 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:16:17.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 120259084312 -lt 120259084311 2024-09-16T08:16:17.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:16:17.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:16:17.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:16:17.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:16:17.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:16:17.777 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:16:17.777 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:16:17.777 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:16:17.777 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:16:17.777 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:16:17.777 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:16:18.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:16:18.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:16:18.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:16:18.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:16:18.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:16:18.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:16:18.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:16:18.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-16T08:16:18.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-16T08:16:19.558 INFO:tasks.workunit.client.0.smithi162.stderr:noout is unset 2024-09-16T08:16:19.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79228-1-10 2024-09-16T08:16:19.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T08:16:19.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T08:16:19.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79228-1-10 2024-09-16T08:16:19.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-16T08:16:19.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-16T08:16:19.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-79228-1-10 td/test-erasure-eio/COPY 2024-09-16T08:16:19.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-16T08:16:19.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-16T08:16:19.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 1 + 1 2024-09-16T08:16:19.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=2 2024-09-16T08:16:19.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-79228-1-10 td/test-erasure-eio 2 10 set 2024-09-16T08:16:19.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79228-1-10 2024-09-16T08:16:19.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-16T08:16:19.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-16T08:16:19.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-16T08:16:19.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=2 2024-09-16T08:16:19.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-16T08:16:19.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-09-16T08:16:19.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-16T08:16:19.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-09-16T08:16:19.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-16T08:16:19.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:16:19.755 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79228-1-10 2024-09-16T08:16:19.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:16:19.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79228-1-10 2024-09-16T08:16:19.756 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79228-1-10 2024-09-16T08:16:19.756 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:16:20.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:16:20.248 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:16:20.248 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T08:16:20.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-16T08:16:20.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-16T08:16:20.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=2 2024-09-16T08:16:20.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-16T08:16:21.023 INFO:tasks.workunit.client.0.smithi162.stderr:noout is set 2024-09-16T08:16:21.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-09-16T08:16:21.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-09-16T08:16:21.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-09-16T08:16:21.038 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records in 2024-09-16T08:16:21.038 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records out 2024-09-16T08:16:21.039 INFO:tasks.workunit.client.0.smithi162.stderr:10 bytes copied, 0.00019272 s, 51.9 kB/s 2024-09-16T08:16:21.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-79228-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:16:21.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-16T08:16:21.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T08:16:21.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-16T08:16:21.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T08:16:21.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-79228-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:16:21.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-16T08:16:21.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T08:16:21.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-16T08:16:21.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T08:16:21.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-16T08:16:21.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:16:21.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:16:21.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:16:21.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:16:21.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:16:21.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:16:21.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-79228-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:16:21.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-16T08:16:21.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T08:16:21.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-16T08:16:21.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T08:16:21.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-16T08:16:21.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-79228-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:16:24.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-16T08:16:24.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-16T08:16:24.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T08:16:24.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T08:16:24.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T08:16:24.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-16T08:16:24.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:16:24.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:16:24.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:16:24.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:16:24.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-16T08:16:24.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-16T08:16:24.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T08:16:24.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T08:16:24.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:16:24.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T08:16:24.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:16:24.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:16:24.202 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:16:24.203 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:16:24.203 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:16:24.203 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:16:24.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:16:24.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:16:24.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:16:24.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:16:24.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:16:24.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:16:24.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:16:24.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:16:24.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T08:16:24.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T08:16:24.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-16T08:16:24.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T08:16:24.204 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.2 2024-09-16T08:16:24.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:16:24.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-16T08:16:24.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T08:16:24.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T08:16:24.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T08:16:24.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T08:16:24.241 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:16:24.238+0000 7f38ef418880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:16:24.246 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:16:24.246+0000 7f38ef418880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:16:24.249 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:16:24.246+0000 7f38ef418880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:16:24.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T08:16:24.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:16:24.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T08:16:24.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:16:24.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:16:24.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:16:24.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:16:24.643 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T08:16:24.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:16:24.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:16:24.796 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:16:24.794+0000 7f38ef418880 -1 Falling back to public interface 2024-09-16T08:16:25.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:16:26.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:16:26.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:16:26.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:16:26.116 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:16:26.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:16:26.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:16:26.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:16:27.416 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:16:27.413+0000 7f38ef418880 -1 osd.2 46 log_to_monitors true 2024-09-16T08:16:27.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:16:27.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:16:27.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:16:27.532 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:16:27.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:16:27.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:16:28.122 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 up in weight 1 up_from 53 up_thru 38 down_at 49 last_clean_interval [19,46) [v2:127.0.0.1:6818/608715526,v1:127.0.0.1:6819/608715526] [v2:127.0.0.1:6820/608715526,v1:127.0.0.1:6821/608715526] exists,up 1ea0584f-efa7-44ae-95cb-2a006c2ccdf8 2024-09-16T08:16:28.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:16:28.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:16:28.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:16:28.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-16T08:16:28.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:16:28.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:16:28.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:16:28.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:16:28.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:16:28.124 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:16:28.124 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:16:28.125 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:16:28.125 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:16:28.125 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:16:28.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:16:28.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:16:28.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:16:28.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:16:28.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:16:28.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:16:28.871 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:16:28.871 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:16:28.871 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:16:28.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:16:28.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:16:28.872 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:16:29.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803822 2024-09-16T08:16:29.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803822 2024-09-16T08:16:29.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803822' 2024-09-16T08:16:29.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:16:29.031 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:16:29.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=180388626440 2024-09-16T08:16:29.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 180388626440 2024-09-16T08:16:29.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803822 1-180388626440' 2024-09-16T08:16:29.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:16:29.180 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:16:29.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=227633266691 2024-09-16T08:16:29.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 227633266691 2024-09-16T08:16:29.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803822 1-180388626440 2-227633266691' 2024-09-16T08:16:29.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:16:29.324 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:16:29.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=120259084316 2024-09-16T08:16:29.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 120259084316 2024-09-16T08:16:29.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803822 1-180388626440 2-227633266691 3-120259084316' 2024-09-16T08:16:29.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:16:29.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803822 2024-09-16T08:16:29.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:16:29.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:16:29.487 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803822 2024-09-16T08:16:29.488 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:16:29.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803822 2024-09-16T08:16:29.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803822' 2024-09-16T08:16:29.490 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803822 2024-09-16T08:16:29.490 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:16:29.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803820 -lt 25769803822 2024-09-16T08:16:29.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T08:16:30.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T08:16:30.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:16:31.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803822 -lt 25769803822 2024-09-16T08:16:31.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:16:31.337 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-180388626440 2024-09-16T08:16:31.337 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:16:31.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:16:31.341 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-180388626440 2024-09-16T08:16:31.341 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:16:31.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=180388626440 2024-09-16T08:16:31.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 180388626440' 2024-09-16T08:16:31.343 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 180388626440 2024-09-16T08:16:31.344 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:16:31.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 180388626440 -lt 180388626440 2024-09-16T08:16:31.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:16:31.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-227633266691 2024-09-16T08:16:31.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:16:31.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:16:31.840 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-227633266691 2024-09-16T08:16:31.840 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:16:31.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=227633266691 2024-09-16T08:16:31.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 227633266691' 2024-09-16T08:16:31.841 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 227633266691 2024-09-16T08:16:31.842 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:16:32.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 227633266691 -lt 227633266691 2024-09-16T08:16:32.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:16:32.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-120259084316 2024-09-16T08:16:32.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:16:32.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:16:32.302 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-120259084316 2024-09-16T08:16:32.302 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:16:32.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=120259084316 2024-09-16T08:16:32.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 120259084316' 2024-09-16T08:16:32.305 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 120259084316 2024-09-16T08:16:32.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:16:32.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 120259084317 -lt 120259084316 2024-09-16T08:16:32.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:16:32.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:16:32.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:16:33.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:16:33.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:16:33.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:16:33.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:16:33.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:16:33.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:16:33.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:16:33.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:16:33.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:16:33.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:16:33.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:16:33.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:16:34.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:16:34.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:16:34.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:16:34.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-16T08:16:34.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-16T08:16:35.092 INFO:tasks.workunit.client.0.smithi162.stderr:noout is unset 2024-09-16T08:16:35.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79228-1-10 fail 2024-09-16T08:16:35.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T08:16:35.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T08:16:35.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79228-1-10 2024-09-16T08:16:35.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-16T08:16:35.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-16T08:16:35.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-79228-1-10 td/test-erasure-eio/COPY 2024-09-16T08:16:35.395 INFO:tasks.workunit.client.0.smithi162.stderr:error getting pool-jerasure/obj-size-79228-1-10: (5) Input/output error 2024-09-16T08:16:35.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-16T08:16:35.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-16T08:16:35.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:324: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 0 2024-09-16T08:16:35.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-16T08:16:35.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-09-16T08:16:35.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=1 2024-09-16T08:16:35.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-09-16T08:16:35.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=0 2024-09-16T08:16:35.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-09-16T08:16:35.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-09-16T08:16:35.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-16T08:16:35.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-79228-1-0 2024-09-16T08:16:35.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-79228-1-0 2024-09-16T08:16:35.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-16T08:16:35.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-16T08:16:35.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-79228-1-0 2024-09-16T08:16:35.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:16:35.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-16T08:16:35.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:16:35.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-16T08:16:35.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:16:35.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-16T08:16:35.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:16:35.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-16T08:16:35.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-79228-1-0 td/test-erasure-eio/ORIGINAL 2024-09-16T08:16:35.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-79228-1-0 td/test-erasure-eio 1 0 set 2024-09-16T08:16:35.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79228-1-0 2024-09-16T08:16:35.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-16T08:16:35.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-16T08:16:35.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-16T08:16:35.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-09-16T08:16:35.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-16T08:16:35.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2024-09-16T08:16:35.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-16T08:16:35.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-09-16T08:16:35.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-16T08:16:35.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:16:35.571 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79228-1-0 2024-09-16T08:16:35.571 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:16:35.571 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79228-1-0 2024-09-16T08:16:35.571 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79228-1-0 2024-09-16T08:16:35.571 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:16:36.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:16:36.047 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:16:36.047 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T08:16:36.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-16T08:16:36.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-16T08:16:36.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-09-16T08:16:36.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-16T08:16:37.012 INFO:tasks.workunit.client.0.smithi162.stderr:noout is set 2024-09-16T08:16:37.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-09-16T08:16:37.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2024-09-16T08:16:37.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2024-09-16T08:16:37.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-79228-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:16:37.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-16T08:16:37.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T08:16:37.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-16T08:16:37.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T08:16:37.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-79228-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:16:37.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-16T08:16:37.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T08:16:37.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-16T08:16:37.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T08:16:37.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-16T08:16:37.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:16:37.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:16:37.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:16:37.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:16:37.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:16:37.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:16:37.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-79228-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:16:37.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-16T08:16:37.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T08:16:37.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-16T08:16:37.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T08:16:37.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-16T08:16:37.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-79228-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:16:39.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-16T08:16:39.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-16T08:16:39.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T08:16:39.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T08:16:39.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T08:16:39.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-16T08:16:39.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:16:39.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:16:39.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:16:39.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:16:39.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-16T08:16:39.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-16T08:16:39.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T08:16:39.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T08:16:39.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:16:39.845 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T08:16:39.845 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:16:39.845 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:16:39.847 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:16:39.847 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:16:39.847 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:16:39.847 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:16:39.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:16:39.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:16:39.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:16:39.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:16:39.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:16:39.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:16:39.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:16:39.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:16:39.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T08:16:39.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T08:16:39.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-16T08:16:39.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T08:16:39.848 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-09-16T08:16:39.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:16:39.849 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-16T08:16:39.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T08:16:39.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T08:16:39.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T08:16:39.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T08:16:39.886 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:16:39.885+0000 7f4f72d0e880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:16:39.890 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:16:39.889+0000 7f4f72d0e880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:16:39.891 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:16:39.889+0000 7f4f72d0e880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:16:40.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T08:16:40.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:16:40.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T08:16:40.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:16:40.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:16:40.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:16:40.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:16:40.274 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T08:16:40.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:16:40.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:16:40.452 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:16:40.449+0000 7f4f72d0e880 -1 Falling back to public interface 2024-09-16T08:16:40.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:16:41.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:16:41.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:16:41.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:16:41.710 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:16:41.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:16:41.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:16:42.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:16:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:16:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:16:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:16:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:16:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:16:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:16:43.247 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:16:43.245+0000 7f4f72d0e880 -1 osd.1 58 log_to_monitors true 2024-09-16T08:16:43.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:16:44.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:16:44.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:16:44.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:16:44.596 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T08:16:44.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:16:44.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:16:45.080 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 63 up_thru 63 down_at 59 last_clean_interval [42,58) [v2:127.0.0.1:6810/3914566567,v1:127.0.0.1:6811/3914566567] [v2:127.0.0.1:6812/3914566567,v1:127.0.0.1:6813/3914566567] exists,up 58db0e1e-7376-4e4a-85b3-d7201fbaf42d 2024-09-16T08:16:45.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:16:45.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:16:45.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:16:45.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-16T08:16:45.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:16:45.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:16:45.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:16:45.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:16:45.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:16:45.081 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:16:45.081 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:16:45.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:16:45.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:16:45.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:16:45.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:16:45.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:16:45.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:16:45.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:16:45.287 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:16:45.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:16:45.778 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:16:45.778 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:16:45.778 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:16:45.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:16:45.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:16:45.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:16:45.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803827 2024-09-16T08:16:45.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803827 2024-09-16T08:16:45.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803827' 2024-09-16T08:16:45.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:16:45.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:16:46.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=270582939651 2024-09-16T08:16:46.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 270582939651 2024-09-16T08:16:46.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803827 1-270582939651' 2024-09-16T08:16:46.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:16:46.100 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:16:46.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=227633266696 2024-09-16T08:16:46.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 227633266696 2024-09-16T08:16:46.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803827 1-270582939651 2-227633266696' 2024-09-16T08:16:46.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:16:46.251 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:16:46.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=120259084322 2024-09-16T08:16:46.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 120259084322 2024-09-16T08:16:46.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803827 1-270582939651 2-227633266696 3-120259084322' 2024-09-16T08:16:46.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:16:46.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803827 2024-09-16T08:16:46.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:16:46.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:16:46.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803827 2024-09-16T08:16:46.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:16:46.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803827 2024-09-16T08:16:46.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803827' 2024-09-16T08:16:46.407 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803827 2024-09-16T08:16:46.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:16:46.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803827 -lt 25769803827 2024-09-16T08:16:46.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:16:46.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-270582939651 2024-09-16T08:16:46.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:16:46.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:16:46.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-270582939651 2024-09-16T08:16:46.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:16:46.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=270582939651 2024-09-16T08:16:46.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 270582939651' 2024-09-16T08:16:46.894 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 270582939651 2024-09-16T08:16:46.894 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:16:47.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 270582939651 -lt 270582939651 2024-09-16T08:16:47.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:16:47.374 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-227633266696 2024-09-16T08:16:47.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:16:47.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:16:47.378 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-227633266696 2024-09-16T08:16:47.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:16:47.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=227633266696 2024-09-16T08:16:47.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 227633266696' 2024-09-16T08:16:47.381 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 227633266696 2024-09-16T08:16:47.382 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:16:47.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 227633266696 -lt 227633266696 2024-09-16T08:16:47.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:16:47.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-120259084322 2024-09-16T08:16:47.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:16:47.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:16:47.776 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-120259084322 2024-09-16T08:16:47.777 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:16:47.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=120259084322 2024-09-16T08:16:47.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 120259084322' 2024-09-16T08:16:47.778 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 120259084322 2024-09-16T08:16:47.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:16:48.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 120259084320 -lt 120259084322 2024-09-16T08:16:48.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T08:16:49.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T08:16:49.234 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:16:49.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 120259084322 -lt 120259084322 2024-09-16T08:16:49.710 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:16:49.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:16:49.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:16:50.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:16:50.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:16:50.339 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:16:50.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:16:50.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:16:50.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:16:50.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:16:50.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:16:50.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:16:50.804 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:16:50.804 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:16:50.805 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:16:51.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:16:51.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:16:51.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:16:51.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-16T08:16:51.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-16T08:16:52.140 INFO:tasks.workunit.client.0.smithi162.stderr:noout is unset 2024-09-16T08:16:52.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79228-1-0 2024-09-16T08:16:52.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T08:16:52.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T08:16:52.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79228-1-0 2024-09-16T08:16:52.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-16T08:16:52.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-16T08:16:52.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-79228-1-0 td/test-erasure-eio/COPY 2024-09-16T08:16:52.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-16T08:16:52.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-16T08:16:52.401 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 1 + 1 2024-09-16T08:16:52.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=2 2024-09-16T08:16:52.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-79228-1-0 td/test-erasure-eio 2 0 set 2024-09-16T08:16:52.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79228-1-0 2024-09-16T08:16:52.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-16T08:16:52.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-16T08:16:52.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-16T08:16:52.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=2 2024-09-16T08:16:52.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-16T08:16:52.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2024-09-16T08:16:52.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-16T08:16:52.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-09-16T08:16:52.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-16T08:16:52.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:16:52.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79228-1-0 2024-09-16T08:16:52.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:16:52.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79228-1-0 2024-09-16T08:16:52.406 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79228-1-0 2024-09-16T08:16:52.406 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:16:52.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:16:52.833 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:16:52.833 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T08:16:52.834 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-16T08:16:52.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-16T08:16:52.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=2 2024-09-16T08:16:52.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-16T08:16:53.707 INFO:tasks.workunit.client.0.smithi162.stderr:noout is set 2024-09-16T08:16:53.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-09-16T08:16:53.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2024-09-16T08:16:53.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2024-09-16T08:16:53.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-79228-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:16:53.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-16T08:16:53.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T08:16:53.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-16T08:16:53.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T08:16:53.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-79228-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:16:53.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-16T08:16:53.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T08:16:53.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-16T08:16:53.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T08:16:53.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-16T08:16:53.726 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:16:53.726 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:16:53.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:16:53.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:16:53.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:16:55.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:16:55.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-79228-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:16:55.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-16T08:16:55.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T08:16:55.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-16T08:16:55.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T08:16:55.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-16T08:16:55.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-79228-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:16:56.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-16T08:16:56.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-16T08:16:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T08:16:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T08:16:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T08:16:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-16T08:16:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:16:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:16:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:16:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:16:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-16T08:16:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-16T08:16:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T08:16:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T08:16:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:16:56.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T08:16:56.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:16:56.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:16:56.749 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:16:56.749 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:16:56.750 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:16:56.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:16:56.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:16:56.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:16:56.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:16:56.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:16:56.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:16:56.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:16:56.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:16:56.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:16:56.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T08:16:56.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T08:16:56.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-16T08:16:56.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T08:16:56.753 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.2 2024-09-16T08:16:56.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:16:56.754 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-16T08:16:56.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T08:16:56.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T08:16:56.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T08:16:56.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T08:16:56.808 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:16:56.805+0000 7fcb87511880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:16:56.811 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:16:56.809+0000 7fcb87511880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:16:56.813 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:16:56.809+0000 7fcb87511880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:16:57.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T08:16:57.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:16:57.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T08:16:57.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:16:57.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:16:57.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:16:57.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:16:57.199 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T08:16:57.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:16:57.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:16:57.355 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:16:57.353+0000 7fcb87511880 -1 Falling back to public interface 2024-09-16T08:16:57.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:16:58.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:16:58.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:16:58.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:16:58.617 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:16:58.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:16:58.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:16:59.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:16:59.792 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:16:59.789+0000 7fcb87511880 -1 osd.2 67 log_to_monitors true 2024-09-16T08:17:00.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:17:00.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:17:00.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:17:00.150 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:17:00.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:17:00.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:17:00.596 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 up in weight 1 up_from 73 up_thru 59 down_at 70 last_clean_interval [53,67) [v2:127.0.0.1:6818/2350759487,v1:127.0.0.1:6819/2350759487] [v2:127.0.0.1:6820/2350759487,v1:127.0.0.1:6821/2350759487] exists,up 1ea0584f-efa7-44ae-95cb-2a006c2ccdf8 2024-09-16T08:17:00.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:17:00.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:17:00.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:17:00.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-16T08:17:00.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:17:00.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:17:00.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:17:00.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:17:00.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:17:00.598 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:17:00.598 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:17:00.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:17:00.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:17:00.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:17:00.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:17:00.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:17:00.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:17:00.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:17:00.813 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:17:01.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:17:01.461 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:17:01.461 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:17:01.461 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:17:01.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:17:01.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:17:01.462 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:17:01.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803832 2024-09-16T08:17:01.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803832 2024-09-16T08:17:01.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803832' 2024-09-16T08:17:01.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:17:01.624 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:17:01.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=270582939656 2024-09-16T08:17:01.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 270582939656 2024-09-16T08:17:01.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803832 1-270582939656' 2024-09-16T08:17:01.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:17:01.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:17:01.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=313532612611 2024-09-16T08:17:01.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 313532612611 2024-09-16T08:17:01.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803832 1-270582939656 2-313532612611' 2024-09-16T08:17:01.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:17:01.977 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:17:02.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=120259084327 2024-09-16T08:17:02.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 120259084327 2024-09-16T08:17:02.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803832 1-270582939656 2-313532612611 3-120259084327' 2024-09-16T08:17:02.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:17:02.161 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803832 2024-09-16T08:17:02.161 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:17:02.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:17:02.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803832 2024-09-16T08:17:02.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:17:02.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803832 2024-09-16T08:17:02.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803832' 2024-09-16T08:17:02.166 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803832 2024-09-16T08:17:02.167 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:17:02.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803833 -lt 25769803832 2024-09-16T08:17:02.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:17:02.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-270582939656 2024-09-16T08:17:02.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:17:02.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:17:02.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-270582939656 2024-09-16T08:17:02.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:17:02.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=270582939656 2024-09-16T08:17:02.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 270582939656' 2024-09-16T08:17:02.610 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 270582939656 2024-09-16T08:17:02.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:17:03.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 270582939656 -lt 270582939656 2024-09-16T08:17:03.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:17:03.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-313532612611 2024-09-16T08:17:03.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:17:03.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:17:03.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-313532612611 2024-09-16T08:17:03.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:17:03.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=313532612611 2024-09-16T08:17:03.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 313532612611' 2024-09-16T08:17:03.154 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 313532612611 2024-09-16T08:17:03.155 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:17:03.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 313532612611 -lt 313532612611 2024-09-16T08:17:03.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:17:03.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-120259084327 2024-09-16T08:17:03.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:17:03.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:17:03.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:17:03.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-120259084327 2024-09-16T08:17:03.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=120259084327 2024-09-16T08:17:03.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 120259084327' 2024-09-16T08:17:03.633 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 120259084327 2024-09-16T08:17:03.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:17:04.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 120259084327 -lt 120259084327 2024-09-16T08:17:04.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:17:04.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:17:04.079 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:17:04.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:17:04.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:17:04.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:17:04.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:17:04.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:17:04.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:17:04.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:17:04.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:17:05.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:17:05.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:17:05.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:17:05.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:17:05.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:17:05.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:17:05.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:17:05.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-16T08:17:05.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-16T08:17:06.397 INFO:tasks.workunit.client.0.smithi162.stderr:noout is unset 2024-09-16T08:17:06.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79228-1-0 fail 2024-09-16T08:17:06.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T08:17:06.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T08:17:06.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79228-1-0 2024-09-16T08:17:06.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-16T08:17:06.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-16T08:17:06.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-79228-1-0 td/test-erasure-eio/COPY 2024-09-16T08:17:06.558 INFO:tasks.workunit.client.0.smithi162.stderr:error getting pool-jerasure/obj-size-79228-1-0: (5) Input/output error 2024-09-16T08:17:06.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-16T08:17:06.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-16T08:17:06.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:325: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 256 add 2024-09-16T08:17:06.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-16T08:17:06.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-09-16T08:17:06.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=1 2024-09-16T08:17:06.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-09-16T08:17:06.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=256 2024-09-16T08:17:06.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-09-16T08:17:06.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=add 2024-09-16T08:17:06.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-16T08:17:06.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-79228-1-256 2024-09-16T08:17:06.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-79228-1-256 2024-09-16T08:17:06.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-16T08:17:06.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-16T08:17:06.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-79228-1-256 2024-09-16T08:17:06.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:17:06.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-16T08:17:06.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:17:06.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-16T08:17:06.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:17:06.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-16T08:17:06.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:17:06.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-16T08:17:06.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-79228-1-256 td/test-erasure-eio/ORIGINAL 2024-09-16T08:17:06.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-79228-1-256 td/test-erasure-eio 1 256 add 2024-09-16T08:17:06.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79228-1-256 2024-09-16T08:17:06.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-16T08:17:06.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-16T08:17:06.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-16T08:17:06.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-09-16T08:17:06.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-16T08:17:06.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2024-09-16T08:17:06.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-16T08:17:06.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2024-09-16T08:17:06.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-16T08:17:06.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:17:06.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79228-1-256 2024-09-16T08:17:06.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:17:06.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79228-1-256 2024-09-16T08:17:06.773 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79228-1-256 2024-09-16T08:17:06.773 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:17:07.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:17:07.262 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:17:07.262 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T08:17:07.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-16T08:17:07.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-16T08:17:07.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-09-16T08:17:07.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-16T08:17:08.045 INFO:tasks.workunit.client.0.smithi162.stderr:noout is set 2024-09-16T08:17:08.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2024-09-16T08:17:08.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-79228-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:17:08.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-16T08:17:08.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T08:17:08.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-16T08:17:08.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T08:17:08.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-79228-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:17:08.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-16T08:17:08.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T08:17:08.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-16T08:17:08.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T08:17:08.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-16T08:17:08.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:17:08.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:17:08.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:17:08.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:17:08.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:17:09.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:17:09.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-79228-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:17:09.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-16T08:17:09.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T08:17:09.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-16T08:17:09.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T08:17:09.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-16T08:17:09.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-79228-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:17:11.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-16T08:17:11.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-16T08:17:11.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T08:17:11.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T08:17:11.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T08:17:11.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-16T08:17:11.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:17:11.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:17:11.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:17:11.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:17:11.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-16T08:17:11.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-16T08:17:11.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T08:17:11.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T08:17:11.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:17:11.345 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T08:17:11.345 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:17:11.345 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:17:11.345 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:17:11.345 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:17:11.345 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:17:11.345 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:17:11.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:17:11.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:17:11.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:17:11.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:17:11.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:17:11.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:17:11.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:17:11.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:17:11.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T08:17:11.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T08:17:11.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-16T08:17:11.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T08:17:11.348 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-09-16T08:17:11.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:17:11.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-16T08:17:11.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T08:17:11.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T08:17:11.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T08:17:11.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T08:17:11.386 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:17:11.385+0000 7ff977f1e880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:17:11.389 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:17:11.389+0000 7ff977f1e880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:17:11.397 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:17:11.397+0000 7ff977f1e880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:17:11.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T08:17:11.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:17:11.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T08:17:11.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:17:11.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:17:11.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:17:11.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:17:11.775 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T08:17:11.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:17:11.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:17:11.940 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:17:11.936+0000 7ff977f1e880 -1 Falling back to public interface 2024-09-16T08:17:12.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:17:13.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:17:13.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:17:13.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:17:13.205 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:17:13.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:17:13.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:17:13.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:17:14.429 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:17:14.424+0000 7ff977f1e880 -1 osd.1 79 log_to_monitors true 2024-09-16T08:17:14.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:17:14.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:17:14.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:17:14.662 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:17:14.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:17:14.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:17:15.265 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 84 up_thru 73 down_at 80 last_clean_interval [63,79) [v2:127.0.0.1:6810/3258425452,v1:127.0.0.1:6811/3258425452] [v2:127.0.0.1:6812/3258425452,v1:127.0.0.1:6813/3258425452] exists,up 58db0e1e-7376-4e4a-85b3-d7201fbaf42d 2024-09-16T08:17:15.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:17:15.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:17:15.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:17:15.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-16T08:17:15.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:17:15.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:17:15.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:17:15.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:17:15.266 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:17:15.266 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:17:15.266 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:17:15.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:17:15.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:17:15.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:17:15.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:17:15.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:17:15.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:17:15.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:17:15.474 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:17:16.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:17:16.110 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:17:16.110 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:17:16.110 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:17:16.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:17:16.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:17:16.110 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:17:16.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803837 2024-09-16T08:17:16.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803837 2024-09-16T08:17:16.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803837' 2024-09-16T08:17:16.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:17:16.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:17:16.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=360777252867 2024-09-16T08:17:16.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 360777252867 2024-09-16T08:17:16.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803837 1-360777252867' 2024-09-16T08:17:16.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:17:16.624 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:17:16.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=313532612616 2024-09-16T08:17:16.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 313532612616 2024-09-16T08:17:16.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803837 1-360777252867 2-313532612616' 2024-09-16T08:17:16.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:17:16.797 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:17:16.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=120259084332 2024-09-16T08:17:16.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 120259084332 2024-09-16T08:17:16.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803837 1-360777252867 2-313532612616 3-120259084332' 2024-09-16T08:17:16.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:17:16.949 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803837 2024-09-16T08:17:16.949 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:17:16.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:17:16.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803837 2024-09-16T08:17:16.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:17:16.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803837 2024-09-16T08:17:16.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803837' 2024-09-16T08:17:16.953 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803837 2024-09-16T08:17:16.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:17:17.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803837 -lt 25769803837 2024-09-16T08:17:17.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:17:17.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-360777252867 2024-09-16T08:17:17.386 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:17:17.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:17:17.389 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-360777252867 2024-09-16T08:17:17.389 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:17:17.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=360777252867 2024-09-16T08:17:17.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 360777252867' 2024-09-16T08:17:17.391 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 360777252867 2024-09-16T08:17:17.392 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:17:17.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 1 -lt 360777252867 2024-09-16T08:17:17.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T08:17:18.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T08:17:18.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:17:19.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 360777252867 -lt 360777252867 2024-09-16T08:17:19.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:17:19.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-313532612616 2024-09-16T08:17:19.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:17:19.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:17:19.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:17:19.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-313532612616 2024-09-16T08:17:19.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=313532612616 2024-09-16T08:17:19.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 313532612616' 2024-09-16T08:17:19.264 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 313532612616 2024-09-16T08:17:19.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:17:19.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 313532612616 -lt 313532612616 2024-09-16T08:17:19.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:17:19.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-120259084332 2024-09-16T08:17:19.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:17:19.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:17:19.743 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-120259084332 2024-09-16T08:17:19.744 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:17:19.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=120259084332 2024-09-16T08:17:19.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 120259084332' 2024-09-16T08:17:19.746 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 120259084332 2024-09-16T08:17:19.746 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:17:20.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 120259084332 -lt 120259084332 2024-09-16T08:17:20.202 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:17:20.202 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:17:20.203 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:17:20.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:17:20.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:17:20.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:17:20.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:17:20.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:17:20.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:17:20.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:17:20.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:17:21.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:17:21.251 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:17:21.252 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:17:21.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:17:21.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:17:21.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:17:21.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:17:21.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2024-09-16T08:17:21.794 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records in 2024-09-16T08:17:21.794 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records out 2024-09-16T08:17:21.794 INFO:tasks.workunit.client.0.smithi162.stderr:256 bytes copied, 7.7209e-05 s, 3.3 MB/s 2024-09-16T08:17:21.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-79228-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:17:21.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-16T08:17:21.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T08:17:21.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-16T08:17:21.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T08:17:21.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-79228-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:17:21.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-16T08:17:21.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T08:17:21.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-16T08:17:21.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T08:17:21.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-16T08:17:21.797 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:17:21.797 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:17:21.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:17:21.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:17:21.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:17:22.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:17:22.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-79228-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:17:22.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-16T08:17:22.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T08:17:22.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-16T08:17:22.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T08:17:22.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-16T08:17:22.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-79228-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:17:24.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-16T08:17:24.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-16T08:17:24.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T08:17:24.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T08:17:24.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T08:17:24.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-16T08:17:24.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:17:24.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:17:24.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:17:24.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:17:24.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-16T08:17:24.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-16T08:17:24.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T08:17:24.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T08:17:24.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:17:24.548 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T08:17:24.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:17:24.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:17:24.550 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:17:24.550 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:17:24.550 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:17:24.550 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:17:24.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:17:24.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:17:24.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:17:24.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:17:24.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:17:24.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:17:24.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:17:24.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:17:24.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T08:17:24.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T08:17:24.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-16T08:17:24.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T08:17:24.552 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-09-16T08:17:24.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:17:24.554 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-16T08:17:24.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T08:17:24.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T08:17:24.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T08:17:24.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T08:17:24.590 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:17:24.588+0000 7fa26aaad880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:17:24.593 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:17:24.592+0000 7fa26aaad880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:17:24.596 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:17:24.592+0000 7fa26aaad880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:17:25.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T08:17:25.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:17:25.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T08:17:25.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:17:25.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:17:25.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:17:25.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:17:25.019 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T08:17:25.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:17:25.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:17:25.156 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:17:25.152+0000 7fa26aaad880 -1 Falling back to public interface 2024-09-16T08:17:25.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:17:26.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:17:26.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:17:26.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:17:26.589 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:17:26.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:17:26.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:17:27.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:17:27.777 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:17:27.772+0000 7fa26aaad880 -1 osd.1 86 log_to_monitors true 2024-09-16T08:17:28.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:17:28.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:17:28.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:17:28.072 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:17:28.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:17:28.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:17:28.777 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 91 up_thru 91 down_at 87 last_clean_interval [84,86) [v2:127.0.0.1:6810/191536014,v1:127.0.0.1:6811/191536014] [v2:127.0.0.1:6812/191536014,v1:127.0.0.1:6813/191536014] exists,up 58db0e1e-7376-4e4a-85b3-d7201fbaf42d 2024-09-16T08:17:28.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:17:28.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:17:28.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:17:28.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-16T08:17:28.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:17:28.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:17:28.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:17:28.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:17:28.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:17:28.780 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:17:28.780 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:17:28.780 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:17:28.780 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:17:28.780 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:17:28.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:17:28.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:17:28.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:17:28.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:17:28.995 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:17:29.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:17:29.464 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:17:29.464 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:17:29.464 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:17:29.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:17:29.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:17:29.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:17:29.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803842 2024-09-16T08:17:29.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803842 2024-09-16T08:17:29.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803842' 2024-09-16T08:17:29.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:17:29.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:17:29.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=390842023939 2024-09-16T08:17:29.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 390842023939 2024-09-16T08:17:29.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803842 1-390842023939' 2024-09-16T08:17:29.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:17:29.830 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:17:29.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=313532612620 2024-09-16T08:17:29.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 313532612620 2024-09-16T08:17:29.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803842 1-390842023939 2-313532612620' 2024-09-16T08:17:29.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:17:29.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:17:30.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=120259084336 2024-09-16T08:17:30.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 120259084336 2024-09-16T08:17:30.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803842 1-390842023939 2-313532612620 3-120259084336' 2024-09-16T08:17:30.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:17:30.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803842 2024-09-16T08:17:30.145 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:17:30.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:17:30.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803842 2024-09-16T08:17:30.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:17:30.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803842 2024-09-16T08:17:30.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803842' 2024-09-16T08:17:30.150 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803842 2024-09-16T08:17:30.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:17:30.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803842 -lt 25769803842 2024-09-16T08:17:30.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:17:30.582 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-390842023939 2024-09-16T08:17:30.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:17:30.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:17:30.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-390842023939 2024-09-16T08:17:30.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:17:30.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=390842023939 2024-09-16T08:17:30.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 390842023939' 2024-09-16T08:17:30.588 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 390842023939 2024-09-16T08:17:30.589 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:17:31.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 390842023939 -lt 390842023939 2024-09-16T08:17:31.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:17:31.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-313532612620 2024-09-16T08:17:31.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:17:31.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:17:31.044 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:17:31.044 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-313532612620 2024-09-16T08:17:31.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=313532612620 2024-09-16T08:17:31.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 313532612620' 2024-09-16T08:17:31.046 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 313532612620 2024-09-16T08:17:31.046 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:17:31.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 313532612621 -lt 313532612620 2024-09-16T08:17:31.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:17:31.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-120259084336 2024-09-16T08:17:31.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:17:31.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:17:31.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-120259084336 2024-09-16T08:17:31.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:17:31.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=120259084336 2024-09-16T08:17:31.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 120259084336' 2024-09-16T08:17:31.476 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 120259084336 2024-09-16T08:17:31.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:17:32.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 120259084336 -lt 120259084336 2024-09-16T08:17:32.130 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:17:32.130 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:17:32.131 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:17:32.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:17:32.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:17:32.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:17:32.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:17:32.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:17:32.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:17:32.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:17:32.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:17:33.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:17:33.135 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:17:33.135 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:17:33.135 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:17:33.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:17:33.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:17:33.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:17:33.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-16T08:17:33.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-16T08:17:34.519 INFO:tasks.workunit.client.0.smithi162.stderr:noout is unset 2024-09-16T08:17:34.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79228-1-256 2024-09-16T08:17:34.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T08:17:34.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T08:17:34.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79228-1-256 2024-09-16T08:17:34.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-16T08:17:34.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-16T08:17:34.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-79228-1-256 td/test-erasure-eio/COPY 2024-09-16T08:17:34.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-16T08:17:34.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-16T08:17:34.661 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 1 + 1 2024-09-16T08:17:34.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=2 2024-09-16T08:17:34.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-79228-1-256 td/test-erasure-eio 2 256 add 2024-09-16T08:17:34.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79228-1-256 2024-09-16T08:17:34.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-16T08:17:34.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-16T08:17:34.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-16T08:17:34.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=2 2024-09-16T08:17:34.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-16T08:17:34.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2024-09-16T08:17:34.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-16T08:17:34.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2024-09-16T08:17:34.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-16T08:17:34.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:17:34.664 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79228-1-256 2024-09-16T08:17:34.664 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:17:34.665 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79228-1-256 2024-09-16T08:17:34.665 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79228-1-256 2024-09-16T08:17:34.665 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:17:35.110 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:17:35.111 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:17:35.111 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T08:17:35.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-16T08:17:35.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-16T08:17:35.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=2 2024-09-16T08:17:35.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-16T08:17:35.883 INFO:tasks.workunit.client.0.smithi162.stderr:noout is set 2024-09-16T08:17:35.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2024-09-16T08:17:35.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-79228-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:17:35.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-16T08:17:35.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T08:17:35.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-16T08:17:35.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T08:17:35.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-79228-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:17:35.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-16T08:17:35.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T08:17:35.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-16T08:17:35.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T08:17:35.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-16T08:17:35.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:17:35.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:17:35.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:17:35.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:17:35.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:17:37.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:17:37.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-79228-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:17:37.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-16T08:17:37.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T08:17:37.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-16T08:17:37.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T08:17:37.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-16T08:17:37.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-79228-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:17:38.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-16T08:17:38.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-16T08:17:38.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T08:17:38.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T08:17:38.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T08:17:38.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-16T08:17:38.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:17:38.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:17:38.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:17:38.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:17:38.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-16T08:17:38.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-16T08:17:38.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T08:17:38.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T08:17:38.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:17:38.665 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T08:17:38.665 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:17:38.665 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:17:38.666 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:17:38.667 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:17:38.667 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:17:38.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:17:38.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:17:38.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:17:38.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:17:38.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:17:38.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:17:38.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:17:38.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:17:38.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:17:38.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T08:17:38.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T08:17:38.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-16T08:17:38.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T08:17:38.668 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.2 2024-09-16T08:17:38.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:17:38.670 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-16T08:17:38.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T08:17:38.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T08:17:38.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T08:17:38.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T08:17:38.705 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:17:38.700+0000 7f5457c63880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:17:38.709 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:17:38.704+0000 7f5457c63880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:17:38.710 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:17:38.708+0000 7f5457c63880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:17:39.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T08:17:39.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:17:39.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T08:17:39.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:17:39.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:17:39.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:17:39.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:17:39.104 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T08:17:39.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:17:39.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:17:39.264 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:17:39.260+0000 7f5457c63880 -1 Falling back to public interface 2024-09-16T08:17:39.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:17:40.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:17:40.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:17:40.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:17:40.617 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:17:40.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:17:40.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:17:41.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:17:41.689 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:17:41.684+0000 7f5457c63880 -1 osd.2 95 log_to_monitors true 2024-09-16T08:17:42.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:17:42.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:17:42.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:17:42.048 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:17:42.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:17:42.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:17:42.630 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 up in weight 1 up_from 101 up_thru 87 down_at 98 last_clean_interval [73,95) [v2:127.0.0.1:6818/3174292801,v1:127.0.0.1:6819/3174292801] [v2:127.0.0.1:6820/3174292801,v1:127.0.0.1:6821/3174292801] exists,up 1ea0584f-efa7-44ae-95cb-2a006c2ccdf8 2024-09-16T08:17:42.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:17:42.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:17:42.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:17:42.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-16T08:17:42.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:17:42.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:17:42.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:17:42.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:17:42.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:17:42.632 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:17:42.632 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:17:42.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:17:42.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:17:42.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:17:42.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:17:42.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:17:42.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:17:42.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:17:42.843 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:17:43.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:17:43.353 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:17:43.353 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:17:43.353 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:17:43.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:17:43.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:17:43.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:17:43.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803847 2024-09-16T08:17:43.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803847 2024-09-16T08:17:43.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803847' 2024-09-16T08:17:43.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:17:43.590 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:17:43.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=390842023944 2024-09-16T08:17:43.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 390842023944 2024-09-16T08:17:43.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803847 1-390842023944' 2024-09-16T08:17:43.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:17:43.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:17:43.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=433791696899 2024-09-16T08:17:43.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 433791696899 2024-09-16T08:17:43.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803847 1-390842023944 2-433791696899' 2024-09-16T08:17:43.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:17:43.911 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:17:44.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=120259084341 2024-09-16T08:17:44.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 120259084341 2024-09-16T08:17:44.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803847 1-390842023944 2-433791696899 3-120259084341' 2024-09-16T08:17:44.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:17:44.060 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803847 2024-09-16T08:17:44.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:17:44.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:17:44.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803847 2024-09-16T08:17:44.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:17:44.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803847 2024-09-16T08:17:44.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803847' 2024-09-16T08:17:44.066 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803847 2024-09-16T08:17:44.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:17:44.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803847 -lt 25769803847 2024-09-16T08:17:44.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:17:44.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-390842023944 2024-09-16T08:17:44.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:17:44.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:17:44.529 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-390842023944 2024-09-16T08:17:44.529 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:17:44.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=390842023944 2024-09-16T08:17:44.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 390842023944' 2024-09-16T08:17:44.530 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 390842023944 2024-09-16T08:17:44.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:17:44.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 390842023944 -lt 390842023944 2024-09-16T08:17:44.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:17:44.969 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-433791696899 2024-09-16T08:17:44.969 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:17:44.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:17:44.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-433791696899 2024-09-16T08:17:44.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:17:44.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=433791696899 2024-09-16T08:17:44.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 433791696899' 2024-09-16T08:17:44.976 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 433791696899 2024-09-16T08:17:44.977 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:17:45.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 433791696899 -lt 433791696899 2024-09-16T08:17:45.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:17:45.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-120259084341 2024-09-16T08:17:45.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:17:45.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:17:45.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-120259084341 2024-09-16T08:17:45.432 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:17:45.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=120259084341 2024-09-16T08:17:45.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 120259084341' 2024-09-16T08:17:45.434 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 120259084341 2024-09-16T08:17:45.434 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:17:45.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 120259084341 -lt 120259084341 2024-09-16T08:17:45.964 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:17:45.965 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:17:45.965 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:17:46.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:17:46.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:17:46.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:17:46.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:17:46.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:17:46.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:17:46.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:17:46.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:17:47.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-09-16T08:17:47.008 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:17:47.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:17:47.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:17:47.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-09-16T08:17:47.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-09-16T08:17:47.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T08:17:47.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-09-16T08:17:47.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T08:17:47.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T08:17:47.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T08:17:47.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:17:47.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:17:47.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:17:47.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:17:47.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:17:47.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:17:47.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:17:48.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-09-16T08:17:48.057 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:17:48.058 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:17:48.058 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:17:48.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-09-16T08:17:48.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-09-16T08:17:48.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T08:17:48.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T08:17:48.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T08:17:48.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T08:17:48.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T08:17:48.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T08:17:48.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T08:17:49.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T08:17:49.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T08:17:49.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-09-16T08:17:49.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T08:17:49.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-16T08:17:49.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T08:17:49.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:17:49.423 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:17:49.423 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:17:49.423 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:17:49.423 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:17:49.423 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:17:49.424 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:17:49.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:17:49.829 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:17:49.830 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:17:49.830 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:17:50.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:17:50.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:17:50.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:17:50.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2024-09-16T08:17:50.395 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records in 2024-09-16T08:17:50.395 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records out 2024-09-16T08:17:50.395 INFO:tasks.workunit.client.0.smithi162.stderr:256 bytes copied, 8.1648e-05 s, 3.1 MB/s 2024-09-16T08:17:50.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-79228-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:17:50.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-16T08:17:50.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T08:17:50.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-16T08:17:50.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T08:17:50.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-79228-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:17:50.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-16T08:17:50.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T08:17:50.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-16T08:17:50.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T08:17:50.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-16T08:17:50.396 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:17:50.397 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:17:50.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:17:50.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:17:50.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:17:50.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:17:50.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-79228-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:17:50.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-16T08:17:50.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T08:17:50.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-16T08:17:50.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T08:17:50.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-16T08:17:50.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-79228-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-16T08:17:53.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-16T08:17:53.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-16T08:17:53.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T08:17:53.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T08:17:53.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T08:17:53.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-16T08:17:53.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:17:53.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:17:53.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:17:53.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:17:53.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-16T08:17:53.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-16T08:17:53.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T08:17:53.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T08:17:53.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:17:53.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T08:17:53.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:17:53.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:17:53.175 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:17:53.175 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:17:53.175 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:17:53.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:17:53.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:17:53.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:17:53.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:17:53.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:17:53.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:17:53.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:17:53.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:17:53.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:17:53.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T08:17:53.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T08:17:53.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-16T08:17:53.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T08:17:53.176 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.2 2024-09-16T08:17:53.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:17:53.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-16T08:17:53.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T08:17:53.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T08:17:53.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T08:17:53.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T08:17:53.214 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:17:53.212+0000 7f676fd17880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:17:53.218 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:17:53.216+0000 7f676fd17880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:17:53.220 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:17:53.216+0000 7f676fd17880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:17:53.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T08:17:53.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:17:53.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T08:17:53.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:17:53.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:17:53.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:17:53.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:17:53.611 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T08:17:53.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:17:53.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:17:53.780 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:17:53.776+0000 7f676fd17880 -1 Falling back to public interface 2024-09-16T08:17:54.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:17:55.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:17:55.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:17:55.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:17:55.055 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:17:55.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:17:55.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:17:55.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:17:56.427 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:17:56.424+0000 7f676fd17880 -1 osd.2 102 log_to_monitors true 2024-09-16T08:17:56.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:17:56.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:17:56.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:17:56.480 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:17:56.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:17:56.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:17:57.064 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 up in weight 1 up_from 107 up_thru 87 down_at 103 last_clean_interval [101,102) [v2:127.0.0.1:6818/2569692106,v1:127.0.0.1:6819/2569692106] [v2:127.0.0.1:6820/2569692106,v1:127.0.0.1:6821/2569692106] exists,up 1ea0584f-efa7-44ae-95cb-2a006c2ccdf8 2024-09-16T08:17:57.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:17:57.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:17:57.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:17:57.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-16T08:17:57.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:17:57.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:17:57.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:17:57.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:17:57.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:17:57.065 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:17:57.066 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:17:57.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:17:57.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:17:57.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:17:57.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:17:57.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:17:57.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:17:57.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:17:57.284 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:17:57.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:17:57.877 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:17:57.878 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:17:57.878 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:17:57.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:17:57.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:17:57.878 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:17:58.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803852 2024-09-16T08:17:58.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803852 2024-09-16T08:17:58.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803852' 2024-09-16T08:17:58.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:17:58.035 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:17:58.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=390842023949 2024-09-16T08:17:58.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 390842023949 2024-09-16T08:17:58.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803852 1-390842023949' 2024-09-16T08:17:58.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:17:58.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:17:58.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=459561500675 2024-09-16T08:17:58.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 459561500675 2024-09-16T08:17:58.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803852 1-390842023949 2-459561500675' 2024-09-16T08:17:58.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:17:58.338 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:17:58.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=120259084346 2024-09-16T08:17:58.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 120259084346 2024-09-16T08:17:58.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803852 1-390842023949 2-459561500675 3-120259084346' 2024-09-16T08:17:58.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:17:58.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803852 2024-09-16T08:17:58.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:17:58.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:17:58.498 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803852 2024-09-16T08:17:58.499 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:17:58.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803852 2024-09-16T08:17:58.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803852' 2024-09-16T08:17:58.501 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803852 2024-09-16T08:17:58.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:17:58.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803852 -lt 25769803852 2024-09-16T08:17:58.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:17:58.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-390842023949 2024-09-16T08:17:58.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:17:58.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:17:58.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-390842023949 2024-09-16T08:17:58.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:17:58.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=390842023949 2024-09-16T08:17:58.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 390842023949' 2024-09-16T08:17:58.935 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 390842023949 2024-09-16T08:17:58.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:17:59.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 390842023949 -lt 390842023949 2024-09-16T08:17:59.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:17:59.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-459561500675 2024-09-16T08:17:59.374 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:17:59.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:17:59.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:17:59.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-459561500675 2024-09-16T08:17:59.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=459561500675 2024-09-16T08:17:59.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 459561500675' 2024-09-16T08:17:59.378 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 459561500675 2024-09-16T08:17:59.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:17:59.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 459561500675 -lt 459561500675 2024-09-16T08:17:59.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:17:59.819 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-120259084346 2024-09-16T08:17:59.819 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:17:59.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:17:59.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-120259084346 2024-09-16T08:17:59.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:17:59.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=120259084346 2024-09-16T08:17:59.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 120259084346' 2024-09-16T08:17:59.826 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 120259084346 2024-09-16T08:17:59.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:18:00.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 120259084344 -lt 120259084346 2024-09-16T08:18:00.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T08:18:01.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T08:18:01.274 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:18:01.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 120259084346 -lt 120259084346 2024-09-16T08:18:01.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:18:01.728 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:18:01.728 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:18:02.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:18:02.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:18:02.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:18:02.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:18:02.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:18:02.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:18:02.274 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:18:02.274 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:18:02.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:18:02.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:18:02.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:18:02.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:18:03.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:18:03.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:18:03.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:18:03.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-16T08:18:03.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-16T08:18:04.037 INFO:tasks.workunit.client.0.smithi162.stderr:noout is unset 2024-09-16T08:18:04.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79228-1-256 fail 2024-09-16T08:18:04.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T08:18:04.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T08:18:04.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79228-1-256 2024-09-16T08:18:04.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-16T08:18:04.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-16T08:18:04.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-79228-1-256 td/test-erasure-eio/COPY 2024-09-16T08:18:04.158 INFO:tasks.workunit.client.0.smithi162.stderr:error getting pool-jerasure/obj-size-79228-1-256: (5) Input/output error 2024-09-16T08:18:04.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-16T08:18:04.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-16T08:18:04.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:326: TEST_rados_get_bad_size_shard_1: delete_erasure_coded_pool pool-jerasure 2024-09-16T08:18:04.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T08:18:04.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-16T08:18:04.815 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-16T08:18:04.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-16T08:18:05.627 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-16T08:18:05.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-16T08:18:05.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T08:18:05.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T08:18:05.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T08:18:05.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:18:05.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:18:05.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:18:05.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:18:05.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:18:05.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:18:05.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T08:18:05.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T08:18:05.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T08:18:05.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T08:18:05.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T08:18:05.964 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T08:18:05.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:18:05.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T08:18:05.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T08:18:05.968 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:18:05.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:18:05.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T08:18:05.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T08:18:06.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T08:18:06.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:18:06.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:18:06.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T08:18:06.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T08:18:06.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T08:18:06.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-16T08:18:06.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-16T08:18:06.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-16T08:18:06.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-16T08:18:06.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T08:18:06.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T08:18:06.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T08:18:06.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:18:06.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:18:06.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:18:06.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:18:06.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:18:06.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:18:06.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T08:18:06.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T08:18:06.017 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T08:18:06.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T08:18:06.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T08:18:06.019 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T08:18:06.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:18:06.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T08:18:06.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T08:18:06.022 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:18:06.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:18:06.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T08:18:06.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T08:18:06.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T08:18:06.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:18:06.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:18:06.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T08:18:06.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T08:18:06.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T08:18:06.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-16T08:18:06.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T08:18:06.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:18:06.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:18:06.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79228 2024-09-16T08:18:06.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T08:18:06.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T08:18:06.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T08:18:06.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-16T08:18:06.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-16T08:18:06.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-16T08:18:06.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T08:18:06.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T08:18:06.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T08:18:06.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-16T08:18:06.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-16T08:18:06.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T08:18:06.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:18:06.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:18:06.464 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:18:06.464 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:18:06.464 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:18:06.464 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:18:06.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T08:18:06.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T08:18:06.704 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T08:18:06.704 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T08:18:06.704 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T08:18:06.704 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T08:18:06.705 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T08:18:06.705 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T08:18:06.705 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T08:18:06.706 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T08:18:06.710 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:18:06.710 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:18:06.710 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:18:06.711 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T08:18:06.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T08:18:06.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get fsid 2024-09-16T08:18:06.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T08:18:06.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T08:18:06.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T08:18:06.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T08:18:06.821 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T08:18:06.821 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T08:18:06.821 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T08:18:06.821 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T08:18:06.822 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:18:06.822 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:18:06.822 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:18:06.822 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T08:18:06.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T08:18:06.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get mon_host 2024-09-16T08:18:06.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-16T08:18:06.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-16T08:18:06.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T08:18:06.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T08:18:06.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T08:18:06.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-16T08:18:06.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T08:18:07.840 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T08:18:07.840 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:18:07.840 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:18:07.841 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:18:07.841 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:18:07.841 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:18:07.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:18:07.843 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T08:18:07.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T08:18:07.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-16T08:18:07.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T08:18:08.247 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-16T08:18:08.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T08:18:09.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-16T08:18:09.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T08:18:09.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T08:18:09.266 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:18:09.266 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:18:09.266 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:18:09.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T08:18:09.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-16T08:18:09.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-mon.a.asok log flush 2024-09-16T08:18:09.352 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T08:18:09.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-16T08:18:09.364 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T08:18:06.515+0000 7f8a222fbd40 0 load: jerasure load: lrc 2024-09-16T08:18:09.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_eio_shard_0 td/test-erasure-eio 2024-09-16T08:18:09.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:255: TEST_rados_get_subread_eio_shard_0: local dir=td/test-erasure-eio 2024-09-16T08:18:09.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:256: TEST_rados_get_subread_eio_shard_0: setup_osds 4 2024-09-16T08:18:09.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-16T08:18:09.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-16T08:18:09.366 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-16T08:18:09.367 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-16T08:18:09.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:18:09.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-16T08:18:09.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:18:09.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:18:09.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T08:18:09.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:18:09.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-16T08:18:09.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:18:09.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:18:09.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:18:09.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:18:09.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-16T08:18:09.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-16T08:18:09.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:18:09.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:18:09.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:18:09.372 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:18:09.372 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:18:09.372 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:18:09.372 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:18:09.373 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:18:09.373 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:18:09.374 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:18:09.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:18:09.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:18:09.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:18:09.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:18:09.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:18:09.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:18:09.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:18:09.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:18:09.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:18:09.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:18:09.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:18:09.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:18:09.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-16T08:18:09.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:18:09.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ccaf2b26-36a3-411b-ab92-075b214784ed 2024-09-16T08:18:09.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ccaf2b26-36a3-411b-ab92-075b214784ed' 2024-09-16T08:18:09.377 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 ccaf2b26-36a3-411b-ab92-075b214784ed 2024-09-16T08:18:09.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:18:09.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDB6edmNUcJGBAAs3jFdyKjeo3CzEZ0leZIfQ== 2024-09-16T08:18:09.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDB6edmNUcJGBAAs3jFdyKjeo3CzEZ0leZIfQ=="}' 2024-09-16T08:18:09.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ccaf2b26-36a3-411b-ab92-075b214784ed -i td/test-erasure-eio/0/new.json 2024-09-16T08:18:09.791 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:18:09.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-16T08:18:09.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDB6edmNUcJGBAAs3jFdyKjeo3CzEZ0leZIfQ== --osd-uuid ccaf2b26-36a3-411b-ab92-075b214784ed 2024-09-16T08:18:09.846 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:18:09.843+0000 7fbb615f0880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:18:09.881 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:18:09.875+0000 7fbb615f0880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:18:09.883 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:18:09.879+0000 7fbb615f0880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:18:09.885 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:18:09.879+0000 7fbb615f0880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T08:18:09.885 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:18:09.879+0000 7fbb615f0880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T08:18:09.885 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:18:09.879+0000 7fbb615f0880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T08:18:09.965 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:18:09.959+0000 7fbb615f0880 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-16T08:18:15.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-16T08:18:15.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:18:15.633 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T08:18:15.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T08:18:15.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:18:16.189 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T08:18:16.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T08:18:16.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:18:16.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:18:16.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:18:16.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:18:16.223 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:18:16.219+0000 7f390d8fc880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:18:16.225 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:18:16.223+0000 7f390d8fc880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:18:16.227 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:18:16.223+0000 7f390d8fc880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:18:16.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T08:18:16.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:18:16.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T08:18:16.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:18:16.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:18:16.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:18:16.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:18:16.578 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:18:16.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:18:16.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:18:16.808 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:18:16.803+0000 7f390d8fc880 -1 Falling back to public interface 2024-09-16T08:18:17.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:18:18.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:18:18.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:18:18.019 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:18:18.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:18:18.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:18:18.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:18:18.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:18:18.862 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:18:18.859+0000 7f390d8fc880 -1 osd.0 0 log_to_monitors true 2024-09-16T08:18:19.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:18:19.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:18:19.448 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:18:19.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:18:19.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:18:19.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:18:20.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:18:21.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:18:21.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:18:21.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:18:21.022 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:18:21.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:18:21.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:18:21.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:18:22.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:18:22.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:18:22.527 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:18:22.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:18:22.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:18:22.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:18:22.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:18:23.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:18:23.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:18:23.993 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:18:23.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:18:23.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:18:23.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:18:24.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:18:25.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:18:25.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:18:25.536 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:18:25.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:18:25.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:18:25.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:18:25.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:18:26.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:18:26.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:18:26.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:18:26.993 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:18:26.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:18:26.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:18:27.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:18:28.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:18:28.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:18:28.564 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:18:28.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:18:28.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:18:28.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:18:28.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:18:29.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:18:29.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:18:29.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:18:29.996 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:18:29.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:18:29.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:18:30.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:18:31.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:18:31.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:18:31.535 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:18:31.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:18:31.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:18:31.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:18:32.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:18:33.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:18:33.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:18:33.034 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:18:33.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:18:33.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:18:33.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:18:33.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:18:34.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:18:34.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:18:34.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:18:34.573 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:18:34.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:18:34.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:18:35.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:18:36.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:18:36.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:18:36.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:18:36.599 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:18:36.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:18:36.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:18:37.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:18:38.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:18:38.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:18:38.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:18:38.322 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:18:38.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:18:38.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:18:38.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:18:39.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:18:39.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:18:39.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:18:39.943 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:18:39.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:18:39.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:18:40.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:18:41.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:18:41.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:18:41.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:18:41.766 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:18:41.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:18:41.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:18:42.220 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3908396325,v1:127.0.0.1:6803/3908396325] [v2:127.0.0.1:6804/3908396325,v1:127.0.0.1:6805/3908396325] exists,up ccaf2b26-36a3-411b-ab92-075b214784ed 2024-09-16T08:18:42.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:18:42.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:18:42.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:18:42.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:18:42.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-16T08:18:42.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:18:42.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:18:42.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T08:18:42.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:18:42.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-16T08:18:42.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:18:42.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:18:42.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:18:42.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:18:42.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-16T08:18:42.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-16T08:18:42.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:18:42.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:18:42.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:18:42.223 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:18:42.223 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:18:42.224 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:18:42.225 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:18:42.225 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:18:42.225 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:18:42.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:18:42.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:18:42.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:18:42.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:18:42.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:18:42.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:18:42.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:18:42.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:18:42.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:18:42.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:18:42.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:18:42.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:18:42.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:18:42.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-16T08:18:42.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:18:42.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=035053f9-472b-4e30-b23a-5869eb3b7ed9 2024-09-16T08:18:42.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 035053f9-472b-4e30-b23a-5869eb3b7ed9' 2024-09-16T08:18:42.228 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 035053f9-472b-4e30-b23a-5869eb3b7ed9 2024-09-16T08:18:42.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:18:42.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDi6edmtEAlDxAAOFs/j8CyJGTRw33lHcadZA== 2024-09-16T08:18:42.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDi6edmtEAlDxAAOFs/j8CyJGTRw33lHcadZA=="}' 2024-09-16T08:18:42.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 035053f9-472b-4e30-b23a-5869eb3b7ed9 -i td/test-erasure-eio/1/new.json 2024-09-16T08:18:43.188 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:18:43.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-16T08:18:43.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDi6edmtEAlDxAAOFs/j8CyJGTRw33lHcadZA== --osd-uuid 035053f9-472b-4e30-b23a-5869eb3b7ed9 2024-09-16T08:18:43.243 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:18:43.239+0000 7f7e1e3dd880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:18:43.246 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:18:43.243+0000 7f7e1e3dd880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:18:43.248 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:18:43.243+0000 7f7e1e3dd880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:18:43.249 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:18:43.243+0000 7f7e1e3dd880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T08:18:43.249 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:18:43.243+0000 7f7e1e3dd880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T08:18:43.249 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:18:43.243+0000 7f7e1e3dd880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T08:18:43.309 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:18:43.307+0000 7f7e1e3dd880 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-16T08:18:49.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-16T08:18:49.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:18:49.545 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-16T08:18:49.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T08:18:49.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:18:50.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T08:18:50.105 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-16T08:18:50.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:18:50.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:18:50.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:18:50.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:18:50.139 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:18:50.134+0000 7fdc2aed2880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:18:50.163 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:18:50.158+0000 7fdc2aed2880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:18:50.165 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:18:50.158+0000 7fdc2aed2880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:18:50.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T08:18:50.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:18:50.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T08:18:50.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:18:50.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:18:50.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:18:50.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:18:50.603 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:18:50.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:18:50.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:18:50.715 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:18:50.710+0000 7fdc2aed2880 -1 Falling back to public interface 2024-09-16T08:18:50.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:18:51.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:18:51.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:18:51.995 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:18:51.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:18:51.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:18:51.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:18:52.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:18:53.323 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:18:53.318+0000 7fdc2aed2880 -1 osd.1 0 log_to_monitors true 2024-09-16T08:18:53.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:18:53.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:18:53.411 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:18:53.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:18:53.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:18:53.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:18:53.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:18:54.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:18:54.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:18:54.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:18:54.937 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:18:54.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:18:54.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:18:55.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:18:56.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:18:56.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:18:56.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:18:56.453 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:18:56.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:18:56.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:18:56.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:18:57.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:18:57.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:18:57.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:18:57.880 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:18:57.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:18:57.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:18:58.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:18:59.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:18:59.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:18:59.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:18:59.539 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:18:59.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:18:59.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:19:00.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:01.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:01.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:01.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:19:01.104 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:19:01.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:01.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:19:01.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:02.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:02.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:02.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:19:02.530 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:19:02.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:19:02.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:03.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:04.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:04.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:04.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:19:04.089 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:19:04.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:04.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:19:04.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:05.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:05.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:05.622 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:19:05.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:19:05.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:05.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:19:06.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:07.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:07.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:07.494 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:19:07.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:19:07.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:07.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:19:08.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:09.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:09.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:09.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:19:09.168 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:19:09.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:09.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:19:09.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:10.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:10.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:10.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:19:10.906 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:19:10.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:10.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:19:12.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:13.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:13.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:13.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:19:13.146 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:19:13.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:13.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:19:13.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:14.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:14.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:14.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:19:14.968 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:19:14.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:14.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:19:15.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:16.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:16.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:16.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:19:16.771 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:19:16.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:16.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:19:17.208 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 14 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2590228872,v1:127.0.0.1:6811/2590228872] [v2:127.0.0.1:6812/2590228872,v1:127.0.0.1:6813/2590228872] exists,up 035053f9-472b-4e30-b23a-5869eb3b7ed9 2024-09-16T08:19:17.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:19:17.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:19:17.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:19:17.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:19:17.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-16T08:19:17.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:19:17.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:19:17.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T08:19:17.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:19:17.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-16T08:19:17.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:19:17.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:19:17.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:19:17.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:19:17.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-16T08:19:17.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-16T08:19:17.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:19:17.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:19:17.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:19:17.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:19:17.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:19:17.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:19:17.212 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:19:17.213 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:19:17.213 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:19:17.213 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:19:17.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:19:17.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:19:17.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:19:17.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:19:17.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:19:17.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:19:17.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:19:17.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:19:17.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:19:17.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:19:17.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:19:17.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:19:17.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-16T08:19:17.216 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:19:17.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b72f1c9a-340e-44eb-afd0-611a27ee0089 2024-09-16T08:19:17.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b72f1c9a-340e-44eb-afd0-611a27ee0089' 2024-09-16T08:19:17.216 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 b72f1c9a-340e-44eb-afd0-611a27ee0089 2024-09-16T08:19:17.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:19:17.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAF6udmaDF9DhAAWy1Q33kfeK53aG02aUjvlg== 2024-09-16T08:19:17.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAF6udmaDF9DhAAWy1Q33kfeK53aG02aUjvlg=="}' 2024-09-16T08:19:17.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b72f1c9a-340e-44eb-afd0-611a27ee0089 -i td/test-erasure-eio/2/new.json 2024-09-16T08:19:17.862 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:19:17.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-16T08:19:17.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAF6udmaDF9DhAAWy1Q33kfeK53aG02aUjvlg== --osd-uuid b72f1c9a-340e-44eb-afd0-611a27ee0089 2024-09-16T08:19:17.906 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:19:17.906+0000 7f6228f34880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:19:17.910 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:19:17.910+0000 7f6228f34880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:19:17.912 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:19:17.910+0000 7f6228f34880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:19:17.913 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:19:17.910+0000 7f6228f34880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:19:17.913 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:19:17.910+0000 7f6228f34880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:19:17.913 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:19:17.910+0000 7f6228f34880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:19:17.993 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:19:17.990+0000 7f6228f34880 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-16T08:19:24.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-16T08:19:24.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:19:24.089 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-16T08:19:24.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T08:19:24.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:19:24.663 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-16T08:19:24.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T08:19:24.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:19:24.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:19:24.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:19:24.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:19:24.695 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:19:24.694+0000 7fe72b254880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:19:24.717 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:19:24.714+0000 7fe72b254880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:19:24.718 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:19:24.718+0000 7fe72b254880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:19:25.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T08:19:25.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:19:25.087 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:19:25.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T08:19:25.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:19:25.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:19:25.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:25.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:19:25.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:25.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:19:25.280 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:19:25.278+0000 7fe72b254880 -1 Falling back to public interface 2024-09-16T08:19:25.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:26.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:26.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:26.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:19:26.578 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:19:26.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:26.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:19:27.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:27.894 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:19:27.894+0000 7fe72b254880 -1 osd.2 0 log_to_monitors true 2024-09-16T08:19:28.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:28.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:28.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:19:28.054 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:19:28.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:28.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:19:28.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:29.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:29.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:29.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:19:29.655 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:19:29.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:19:29.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:30.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:31.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:31.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:31.272 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:19:31.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:19:31.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:31.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:19:31.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:32.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:32.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:32.944 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:19:32.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:19:32.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:32.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:19:33.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:34.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:34.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:34.396 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:19:34.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:19:34.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:34.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:19:34.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:35.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:35.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:35.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:19:35.882 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:19:35.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:35.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:19:36.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:37.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:37.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:37.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:19:37.395 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:19:37.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:37.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:19:37.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:38.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:38.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:38.918 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:19:38.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:19:38.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:38.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:19:39.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:40.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:40.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:40.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:19:40.372 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:19:40.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:40.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:19:41.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:42.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:42.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:42.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:19:42.438 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:19:42.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:42.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:19:43.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:44.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:44.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:44.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:19:44.261 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:19:44.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:19:44.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:44.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:45.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:45.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:45.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:19:45.842 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:19:45.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:45.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:19:46.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:47.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:47.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:47.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:19:47.484 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:19:47.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:47.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:19:48.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:49.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:49.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:49.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:19:49.501 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:19:49.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:49.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:19:50.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:19:51.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:19:51.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:19:51.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:19:51.275 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:19:51.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:19:51.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:19:51.899 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1057342652,v1:127.0.0.1:6819/1057342652] [v2:127.0.0.1:6820/1057342652,v1:127.0.0.1:6821/1057342652] exists,up b72f1c9a-340e-44eb-afd0-611a27ee0089 2024-09-16T08:19:51.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:19:51.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:19:51.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:19:51.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:19:51.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-16T08:19:51.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:19:51.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:19:51.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T08:19:51.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:19:51.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-16T08:19:51.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:19:51.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:19:51.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:19:51.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:19:51.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-16T08:19:51.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-16T08:19:51.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:19:51.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:19:51.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:19:51.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:19:51.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:19:51.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:19:51.907 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:19:51.907 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:19:51.907 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:19:51.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:19:51.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:19:51.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:19:51.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:19:51.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:19:51.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:19:51.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:19:51.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:19:51.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:19:51.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:19:51.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:19:51.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:19:51.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:19:51.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-16T08:19:51.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:19:51.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c13de7c5-083e-4835-8769-cb609d1c345e 2024-09-16T08:19:51.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 c13de7c5-083e-4835-8769-cb609d1c345e' 2024-09-16T08:19:51.909 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 c13de7c5-083e-4835-8769-cb609d1c345e 2024-09-16T08:19:51.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:19:51.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAn6udmg8jaNxAAJXCJxSja3AML/NxMcanK4g== 2024-09-16T08:19:51.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAn6udmg8jaNxAAJXCJxSja3AML/NxMcanK4g=="}' 2024-09-16T08:19:51.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c13de7c5-083e-4835-8769-cb609d1c345e -i td/test-erasure-eio/3/new.json 2024-09-16T08:19:52.972 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:19:52.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-16T08:19:52.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAn6udmg8jaNxAAJXCJxSja3AML/NxMcanK4g== --osd-uuid c13de7c5-083e-4835-8769-cb609d1c345e 2024-09-16T08:19:53.027 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:19:53.025+0000 7fcd74468880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:19:53.031 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:19:53.029+0000 7fcd74468880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:19:53.033 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:19:53.029+0000 7fcd74468880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:19:53.033 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:19:53.029+0000 7fcd74468880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:19:53.033 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:19:53.029+0000 7fcd74468880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:19:53.033 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:19:53.029+0000 7fcd74468880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:19:53.127 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:19:53.125+0000 7fcd74468880 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-16T08:19:59.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-16T08:19:59.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:19:59.353 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-16T08:19:59.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T08:19:59.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:19:59.956 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-16T08:19:59.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T08:19:59.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:19:59.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:19:59.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:19:59.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:19:59.991 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:19:59.989+0000 7fc1e93ae880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:20:00.027 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:20:00.025+0000 7fc1e93ae880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:20:00.029 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:20:00.025+0000 7fc1e93ae880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:20:00.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T08:20:00.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:20:00.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T08:20:00.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:20:00.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:20:00.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:20:00.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:20:00.464 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:20:00.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:20:00.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:20:00.587 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:20:00.585+0000 7fc1e93ae880 -1 Falling back to public interface 2024-09-16T08:20:00.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:20:01.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:20:01.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:20:01.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:20:01.888 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:20:01.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:20:01.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:20:02.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:20:03.047 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:20:03.045+0000 7fc1e93ae880 -1 osd.3 0 log_to_monitors true 2024-09-16T08:20:03.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:20:03.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:20:03.414 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:20:03.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:20:03.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:20:03.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:20:03.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:20:04.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:20:04.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:20:04.994 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:20:04.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:20:04.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:20:04.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:20:05.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:20:06.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:20:06.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:20:06.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:20:06.733 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:20:06.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:20:06.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:20:07.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:20:08.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:20:08.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:20:08.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:20:08.308 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:20:08.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:20:08.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:20:08.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:20:09.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:20:09.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:20:09.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:20:09.874 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:20:09.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:20:09.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:20:10.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:20:11.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:20:11.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:20:11.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:20:11.406 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:20:11.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:20:11.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:20:11.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:20:12.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:20:12.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:20:12.985 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:20:12.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:20:12.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:20:12.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:20:13.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:20:14.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:20:14.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:20:14.421 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:20:14.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:20:14.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:20:14.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:20:14.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:20:15.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:20:15.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:20:15.976 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:20:15.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:20:15.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:20:15.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:20:17.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:20:18.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:20:18.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:20:18.027 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:20:18.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:20:18.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:20:18.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:20:18.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:20:19.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:20:19.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:20:19.627 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:20:19.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:20:19.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:20:19.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:20:20.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:20:21.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:20:21.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:20:21.442 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:20:21.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:20:21.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:20:21.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:20:22.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:20:23.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:20:23.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:20:23.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:20:23.077 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:20:23.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:20:23.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:20:23.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:20:24.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:20:24.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:20:24.546 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:20:24.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:20:24.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:20:24.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:20:25.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:20:26.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:20:26.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:20:26.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:20:26.188 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:20:26.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:20:26.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:20:26.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:20:27.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:20:27.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:20:27.732 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:20:27.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:20:27.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:20:27.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:20:28.238 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/763433555,v1:127.0.0.1:6827/763433555] [v2:127.0.0.1:6828/763433555,v1:127.0.0.1:6829/763433555] exists,up c13de7c5-083e-4835-8769-cb609d1c345e 2024-09-16T08:20:28.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:20:28.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:20:28.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:20:28.240 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-16T08:20:28.240 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T08:20:28.240 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T08:20:28.241 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:20:28.241 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:20:28.241 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:20:28.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.0.asok 2024-09-16T08:20:28.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-16T08:20:28.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.0.asok log flush 2024-09-16T08:20:28.329 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T08:20:28.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-16T08:20:28.340 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T08:18:16.819+0000 7f390d8fc880 0 load: jerasure load: lrc 2024-09-16T08:20:28.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:258: TEST_rados_get_subread_eio_shard_0: local poolname=pool-jerasure 2024-09-16T08:20:28.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:259: TEST_rados_get_subread_eio_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2024-09-16T08:20:28.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T08:20:28.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-16T08:20:28.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-16T08:20:28.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-16T08:20:28.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-16T08:20:28.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-16T08:20:28.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-16T08:20:29.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-16T08:20:29.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-16T08:20:30.254 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-16T08:20:30.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T08:20:31.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-16T08:20:31.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:20:31.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:20:31.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:20:31.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:20:31.270 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:20:31.271 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:20:31.271 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:20:31.271 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:20:31.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:20:31.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:20:31.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:20:31.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:20:31.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:20:31.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:20:31.503 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:20:31.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:20:31.947 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:20:31.947 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:20:31.947 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:20:31.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:20:31.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:20:31.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:20:32.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803811 2024-09-16T08:20:32.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803811 2024-09-16T08:20:32.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803811' 2024-09-16T08:20:32.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:20:32.203 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:20:32.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=60129542171 2024-09-16T08:20:32.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 60129542171 2024-09-16T08:20:32.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803811 1-60129542171' 2024-09-16T08:20:32.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:20:32.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:20:32.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378645 2024-09-16T08:20:32.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378645 2024-09-16T08:20:32.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803811 1-60129542171 2-81604378645' 2024-09-16T08:20:32.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:20:32.553 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:20:32.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=120259084306 2024-09-16T08:20:32.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 120259084306 2024-09-16T08:20:32.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803811 1-60129542171 2-81604378645 3-120259084306' 2024-09-16T08:20:32.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:20:32.710 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803811 2024-09-16T08:20:32.710 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:20:32.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:20:32.713 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803811 2024-09-16T08:20:32.713 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:20:32.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803811 2024-09-16T08:20:32.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803811' 2024-09-16T08:20:32.715 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803811 2024-09-16T08:20:32.716 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:20:33.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803809 -lt 25769803811 2024-09-16T08:20:33.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T08:20:34.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T08:20:34.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:20:34.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803812 -lt 25769803811 2024-09-16T08:20:34.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:20:34.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-60129542171 2024-09-16T08:20:34.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:20:34.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:20:34.625 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-60129542171 2024-09-16T08:20:34.625 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:20:34.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=60129542171 2024-09-16T08:20:34.628 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 60129542171 2024-09-16T08:20:34.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 60129542171' 2024-09-16T08:20:34.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:20:35.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 60129542171 -lt 60129542171 2024-09-16T08:20:35.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:20:35.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378645 2024-09-16T08:20:35.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:20:35.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:20:35.084 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:20:35.084 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378645 2024-09-16T08:20:35.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378645 2024-09-16T08:20:35.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378645' 2024-09-16T08:20:35.086 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 81604378645 2024-09-16T08:20:35.086 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:20:35.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378645 -lt 81604378645 2024-09-16T08:20:35.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:20:35.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-120259084306 2024-09-16T08:20:35.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:20:35.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:20:35.543 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-120259084306 2024-09-16T08:20:35.543 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:20:35.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=120259084306 2024-09-16T08:20:35.545 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 120259084306 2024-09-16T08:20:35.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 120259084306' 2024-09-16T08:20:35.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:20:35.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 120259084306 -lt 120259084306 2024-09-16T08:20:35.958 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:20:35.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:20:35.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:20:36.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:20:36.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:20:36.474 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:20:36.474 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:20:36.474 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:20:36.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:20:36.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:20:36.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:20:36.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:20:36.965 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:20:36.966 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:20:36.966 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:20:37.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:20:37.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:20:37.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:20:37.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:261: TEST_rados_get_subread_eio_shard_0: local shard_id=0 2024-09-16T08:20:37.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:262: TEST_rados_get_subread_eio_shard_0: rados_put_get_data eio td/test-erasure-eio 0 2024-09-16T08:20:37.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-09-16T08:20:37.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-16T08:20:37.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-16T08:20:37.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-16T08:20:37.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=0 2024-09-16T08:20:37.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-16T08:20:37.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2024-09-16T08:20:37.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-16T08:20:37.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-79228-0 2024-09-16T08:20:37.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-79228-0 2024-09-16T08:20:37.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-16T08:20:37.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-16T08:20:37.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-79228-0 2024-09-16T08:20:37.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:20:37.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-16T08:20:37.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:20:37.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-16T08:20:37.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:20:37.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-16T08:20:37.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:20:37.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-16T08:20:37.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-79228-0 td/test-erasure-eio/ORIGINAL 2024-09-16T08:20:37.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-79228-0 td/test-erasure-eio 0 2024-09-16T08:20:37.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=ec 2024-09-16T08:20:37.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T08:20:37.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T08:20:37.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T08:20:37.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-jerasure 2024-09-16T08:20:37.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T08:20:37.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-eio-79228-0 2024-09-16T08:20:37.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T08:20:37.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/test-erasure-eio 2024-09-16T08:20:37.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T08:20:37.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T08:20:37.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T08:20:37.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:20:37.744 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-jerasure obj-eio-79228-0 2024-09-16T08:20:37.744 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:20:37.745 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-79228-0 2024-09-16T08:20:37.745 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-79228-0 2024-09-16T08:20:37.745 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:20:38.204 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:20:38.205 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:20:38.205 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T08:20:38.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-16T08:20:38.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T08:20:38.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T08:20:38.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' ec '!=' ec ']' 2024-09-16T08:20:38.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/test-erasure-eio/3/type 2024-09-16T08:20:38.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T08:20:38.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T08:20:38.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T08:20:38.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T08:20:38.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T08:20:38.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T08:20:38.209 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T08:20:38.210 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T08:20:38.210 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T08:20:38.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T08:20:38.210 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:20:38.211 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:20:38.211 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:20:38.211 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.3.asok 2024-09-16T08:20:38.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79228/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T08:20:38.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T08:20:38.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T08:20:38.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T08:20:38.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T08:20:38.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T08:20:38.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T08:20:38.306 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:20:38.306 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:20:38.306 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:20:38.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.3.asok 2024-09-16T08:20:38.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T08:20:38.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.3.asok injectdataerr pool-jerasure obj-eio-79228-0 0 2024-09-16T08:20:38.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-79228-0 2024-09-16T08:20:38.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T08:20:38.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T08:20:38.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-79228-0 2024-09-16T08:20:38.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-16T08:20:38.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-16T08:20:38.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-79228-0 td/test-erasure-eio/COPY 2024-09-16T08:20:38.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-16T08:20:38.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-16T08:20:38.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2024-09-16T08:20:38.453 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 0 + 1 2024-09-16T08:20:38.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: shard_id=1 2024-09-16T08:20:38.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-79228-0 td/test-erasure-eio 1 2024-09-16T08:20:38.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=ec 2024-09-16T08:20:38.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T08:20:38.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T08:20:38.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T08:20:38.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-jerasure 2024-09-16T08:20:38.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T08:20:38.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-eio-79228-0 2024-09-16T08:20:38.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T08:20:38.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/test-erasure-eio 2024-09-16T08:20:38.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T08:20:38.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=1 2024-09-16T08:20:38.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T08:20:38.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:20:38.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-jerasure obj-eio-79228-0 2024-09-16T08:20:38.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:20:38.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-79228-0 2024-09-16T08:20:38.458 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-79228-0 2024-09-16T08:20:38.459 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:20:38.868 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:20:38.868 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:20:38.868 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T08:20:38.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-16T08:20:38.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T08:20:38.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=1 2024-09-16T08:20:38.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' ec '!=' ec ']' 2024-09-16T08:20:38.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/test-erasure-eio/1/type 2024-09-16T08:20:38.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T08:20:38.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-16T08:20:38.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T08:20:38.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-16T08:20:38.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T08:20:38.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T08:20:38.873 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T08:20:38.874 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-16T08:20:38.874 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T08:20:38.874 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T08:20:38.875 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:20:38.875 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:20:38.875 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:20:38.875 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.1.asok 2024-09-16T08:20:38.876 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79228/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-16T08:20:38.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T08:20:38.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T08:20:38.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T08:20:38.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.1 2024-09-16T08:20:38.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T08:20:38.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T08:20:38.972 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:20:38.972 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:20:38.972 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:20:38.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.1.asok 2024-09-16T08:20:38.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T08:20:38.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-79228-0 1 2024-09-16T08:20:39.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-79228-0 fail 2024-09-16T08:20:39.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T08:20:39.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T08:20:39.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-79228-0 2024-09-16T08:20:39.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-16T08:20:39.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-16T08:20:39.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-eio-79228-0 td/test-erasure-eio/COPY 2024-09-16T08:20:39.109 INFO:tasks.workunit.client.0.smithi162.stderr:error getting pool-jerasure/obj-eio-79228-0: (5) Input/output error 2024-09-16T08:20:39.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-16T08:20:39.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:184: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2024-09-16T08:20:39.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:263: TEST_rados_get_subread_eio_shard_0: delete_erasure_coded_pool pool-jerasure 2024-09-16T08:20:39.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T08:20:39.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-16T08:20:39.710 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-16T08:20:39.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-16T08:20:40.554 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-16T08:20:40.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-16T08:20:40.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T08:20:40.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T08:20:40.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T08:20:40.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:20:40.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:20:40.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:20:40.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:20:40.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:20:40.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:20:40.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T08:20:40.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T08:20:40.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T08:20:40.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T08:20:40.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T08:20:40.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T08:20:40.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:20:40.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T08:20:40.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T08:20:40.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:20:40.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:20:40.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T08:20:40.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T08:20:40.919 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T08:20:40.919 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:20:40.920 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:20:40.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T08:20:40.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T08:20:40.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T08:20:40.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-16T08:20:40.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-16T08:20:40.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-16T08:20:40.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-16T08:20:40.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T08:20:40.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T08:20:40.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T08:20:40.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:20:40.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:20:40.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:20:40.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:20:40.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:20:40.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:20:40.930 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T08:20:40.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T08:20:40.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T08:20:40.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T08:20:40.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T08:20:40.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T08:20:40.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:20:40.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T08:20:40.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T08:20:40.937 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:20:40.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:20:40.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T08:20:40.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T08:20:40.943 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T08:20:40.943 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:20:40.943 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:20:40.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T08:20:40.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T08:20:40.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T08:20:40.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-16T08:20:40.948 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T08:20:40.948 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:20:40.948 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:20:40.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79228 2024-09-16T08:20:40.951 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T08:20:40.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T08:20:40.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T08:20:40.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-16T08:20:40.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-16T08:20:40.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-16T08:20:40.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T08:20:40.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T08:20:40.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T08:20:40.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-16T08:20:40.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-16T08:20:41.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T08:20:41.396 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:20:41.396 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:20:41.396 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:20:41.397 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:20:41.397 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:20:41.397 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:20:41.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T08:20:41.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T08:20:41.641 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T08:20:41.641 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T08:20:41.641 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T08:20:41.641 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T08:20:41.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T08:20:41.642 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T08:20:41.642 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T08:20:41.642 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T08:20:41.643 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:20:41.643 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:20:41.643 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:20:41.644 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T08:20:41.645 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T08:20:41.645 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get fsid 2024-09-16T08:20:41.742 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T08:20:41.742 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T08:20:41.742 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T08:20:41.742 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T08:20:41.743 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T08:20:41.743 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T08:20:41.743 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T08:20:41.743 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T08:20:41.744 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:20:41.744 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:20:41.744 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:20:41.744 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T08:20:41.745 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T08:20:41.745 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get mon_host 2024-09-16T08:20:41.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-16T08:20:41.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-16T08:20:41.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T08:20:41.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T08:20:41.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T08:20:41.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-16T08:20:41.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T08:20:42.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T08:20:42.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:20:42.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:20:42.775 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:20:42.775 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:20:42.775 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:20:42.775 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:20:42.776 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T08:20:42.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T08:20:42.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-16T08:20:42.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T08:20:43.176 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-16T08:20:43.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T08:20:44.188 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-16T08:20:44.188 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T08:20:44.188 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T08:20:44.189 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:20:44.189 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:20:44.189 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:20:44.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T08:20:44.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-16T08:20:44.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-mon.a.asok log flush 2024-09-16T08:20:44.271 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T08:20:44.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-16T08:20:44.283 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T08:20:41.448+0000 7f613d27ed40 0 load: jerasure load: lrc 2024-09-16T08:20:44.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_eio_shard_1 td/test-erasure-eio 2024-09-16T08:20:44.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:267: TEST_rados_get_subread_eio_shard_1: local dir=td/test-erasure-eio 2024-09-16T08:20:44.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:268: TEST_rados_get_subread_eio_shard_1: setup_osds 4 2024-09-16T08:20:44.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-16T08:20:44.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-16T08:20:44.285 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-16T08:20:44.286 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-16T08:20:44.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:20:44.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-16T08:20:44.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:20:44.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:20:44.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T08:20:44.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:20:44.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-16T08:20:44.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:20:44.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:20:44.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:20:44.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:20:44.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-16T08:20:44.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-16T08:20:44.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:20:44.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:20:44.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:20:44.294 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:20:44.294 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:20:44.294 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:20:44.294 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:20:44.294 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:20:44.294 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:20:44.294 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:20:44.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:20:44.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:20:44.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:20:44.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:20:44.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:20:44.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:20:44.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:20:44.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:20:44.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:20:44.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:20:44.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:20:44.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:20:44.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-16T08:20:44.295 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:20:44.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ab88145e-b068-4d07-b0b5-7f123d8537ea 2024-09-16T08:20:44.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ab88145e-b068-4d07-b0b5-7f123d8537ea' 2024-09-16T08:20:44.296 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 ab88145e-b068-4d07-b0b5-7f123d8537ea 2024-09-16T08:20:44.296 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:20:44.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBc6udmSvAKExAAuvqCq0X16IfyRiEd4J5rjw== 2024-09-16T08:20:44.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBc6udmSvAKExAAuvqCq0X16IfyRiEd4J5rjw=="}' 2024-09-16T08:20:44.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ab88145e-b068-4d07-b0b5-7f123d8537ea -i td/test-erasure-eio/0/new.json 2024-09-16T08:20:44.678 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:20:44.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-16T08:20:44.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBc6udmSvAKExAAuvqCq0X16IfyRiEd4J5rjw== --osd-uuid ab88145e-b068-4d07-b0b5-7f123d8537ea 2024-09-16T08:20:44.728 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:20:44.724+0000 7f3ca59b0880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:20:44.730 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:20:44.728+0000 7f3ca59b0880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:20:44.732 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:20:44.728+0000 7f3ca59b0880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:20:44.732 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:20:44.728+0000 7f3ca59b0880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T08:20:44.732 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:20:44.728+0000 7f3ca59b0880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T08:20:44.732 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:20:44.728+0000 7f3ca59b0880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T08:20:44.810 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:20:44.808+0000 7f3ca59b0880 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-16T08:20:50.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-16T08:20:50.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:20:50.593 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T08:20:50.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T08:20:50.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:20:51.157 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T08:20:51.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T08:20:51.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:20:51.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:20:51.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:20:51.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:20:51.192 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:20:51.188+0000 7fde01c09880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:20:51.194 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:20:51.192+0000 7fde01c09880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:20:51.198 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:20:51.196+0000 7fde01c09880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:20:51.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T08:20:51.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:20:51.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T08:20:51.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:20:51.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:20:51.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:20:51.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:20:51.574 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:20:51.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:20:51.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:20:51.792 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:20:51.788+0000 7fde01c09880 -1 Falling back to public interface 2024-09-16T08:20:52.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:20:53.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:20:53.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:20:53.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:20:53.028 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:20:53.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:20:53.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:20:53.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:20:54.433 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:20:54.428+0000 7fde01c09880 -1 osd.0 0 log_to_monitors true 2024-09-16T08:20:54.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:20:54.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:20:54.443 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:20:54.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:20:54.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:20:54.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:20:55.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:20:56.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:20:56.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:20:56.029 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:20:56.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:20:56.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:20:56.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:20:56.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:20:57.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:20:57.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:20:57.590 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:20:57.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:20:57.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:20:57.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:20:57.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:20:58.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:20:58.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:20:58.994 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:20:58.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:20:58.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:20:58.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:20:59.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:00.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:00.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:00.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:21:00.483 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:21:00.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:00.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:21:00.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:01.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:01.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:01.944 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:21:01.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:21:01.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:01.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:21:02.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:03.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:03.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:03.497 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:21:03.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:21:03.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:03.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:21:03.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:04.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:04.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:04.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:21:04.954 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:21:04.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:04.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:21:05.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:06.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:06.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:06.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:21:06.501 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:21:06.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:06.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:21:06.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:07.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:07.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:07.939 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:21:07.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:21:07.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:07.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:21:08.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:09.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:09.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:09.925 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:21:09.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:21:09.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:09.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:21:10.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:11.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:11.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:11.771 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:21:11.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:21:11.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:11.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:21:12.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:13.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:13.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:13.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:21:13.740 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:21:13.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:13.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:21:14.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:15.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:15.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:15.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:21:15.829 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:21:15.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:15.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:21:16.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:17.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:17.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:17.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:21:17.593 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:21:17.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:17.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:21:18.189 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1574134268,v1:127.0.0.1:6803/1574134268] [v2:127.0.0.1:6804/1574134268,v1:127.0.0.1:6805/1574134268] exists,up ab88145e-b068-4d07-b0b5-7f123d8537ea 2024-09-16T08:21:18.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:21:18.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:21:18.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:21:18.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:21:18.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-16T08:21:18.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:21:18.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:21:18.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T08:21:18.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:21:18.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-16T08:21:18.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:21:18.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:21:18.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:21:18.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:21:18.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-16T08:21:18.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-16T08:21:18.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:21:18.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:21:18.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:21:18.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:21:18.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:21:18.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:21:18.194 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:21:18.194 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:21:18.194 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:21:18.195 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:21:18.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:21:18.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:21:18.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:21:18.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:21:18.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:21:18.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:21:18.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:21:18.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:21:18.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:21:18.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:21:18.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:21:18.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:21:18.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-16T08:21:18.196 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:21:18.197 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 a6bbe05b-67fa-4e47-aa29-34c357800929 2024-09-16T08:21:18.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a6bbe05b-67fa-4e47-aa29-34c357800929 2024-09-16T08:21:18.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a6bbe05b-67fa-4e47-aa29-34c357800929' 2024-09-16T08:21:18.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:21:18.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB+6udmqXtiDRAAKf39P3r8uVykqk4up6uvRA== 2024-09-16T08:21:18.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB+6udmqXtiDRAAKf39P3r8uVykqk4up6uvRA=="}' 2024-09-16T08:21:18.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a6bbe05b-67fa-4e47-aa29-34c357800929 -i td/test-erasure-eio/1/new.json 2024-09-16T08:21:19.051 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:21:19.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-16T08:21:19.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB+6udmqXtiDRAAKf39P3r8uVykqk4up6uvRA== --osd-uuid a6bbe05b-67fa-4e47-aa29-34c357800929 2024-09-16T08:21:19.107 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:21:19.103+0000 7fce7b59f880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:21:19.149 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:21:19.147+0000 7fce7b59f880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:21:19.151 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:21:19.147+0000 7fce7b59f880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:21:19.152 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:21:19.147+0000 7fce7b59f880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T08:21:19.153 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:21:19.147+0000 7fce7b59f880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T08:21:19.153 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:21:19.147+0000 7fce7b59f880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T08:21:19.240 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:21:19.235+0000 7fce7b59f880 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-16T08:21:25.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-16T08:21:25.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:21:25.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T08:21:25.189 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-16T08:21:25.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:21:25.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T08:21:25.753 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-16T08:21:25.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:21:25.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:21:25.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:21:25.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:21:25.779 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:21:25.775+0000 7fc8f6d29880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:21:25.782 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:21:25.779+0000 7fc8f6d29880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:21:25.789 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:21:25.779+0000 7fc8f6d29880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:21:26.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T08:21:26.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:21:26.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T08:21:26.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:21:26.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:21:26.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:26.251 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:21:26.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:21:26.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:26.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:21:26.324 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:21:26.319+0000 7fc8f6d29880 -1 Falling back to public interface 2024-09-16T08:21:26.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:27.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:27.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:27.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:21:27.657 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:21:27.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:27.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:21:28.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:28.898 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:21:28.895+0000 7fc8f6d29880 -1 osd.1 0 log_to_monitors true 2024-09-16T08:21:29.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:29.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:29.068 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:21:29.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:21:29.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:29.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:21:29.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:30.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:30.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:30.648 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:21:30.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:21:30.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:30.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:21:31.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:32.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:32.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:32.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:21:32.193 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:21:32.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:32.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:21:32.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:33.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:33.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:33.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:21:33.628 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:21:33.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:33.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:21:34.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:35.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:35.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:35.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:21:35.073 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:21:35.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:35.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:21:35.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:36.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:36.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:36.534 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:21:36.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:21:36.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:36.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:21:37.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:38.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:38.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:38.162 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:21:38.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:21:38.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:38.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:21:38.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:39.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:39.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:39.778 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:21:39.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:21:39.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:39.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:21:40.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:41.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:41.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:41.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:21:41.278 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:21:41.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:41.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:21:41.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:42.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:42.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:42.899 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:21:42.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:21:42.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:42.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:21:43.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:44.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:44.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:44.673 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:21:44.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:21:44.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:44.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:21:45.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:46.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:46.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:46.644 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:21:46.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:21:46.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:46.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:21:47.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:48.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:48.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:48.667 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:21:48.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:21:48.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:48.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:21:49.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:50.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:50.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:50.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:21:50.367 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:21:50.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:50.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:21:51.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:21:52.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:21:52.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:21:52.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:21:52.427 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:21:52.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:21:52.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:21:53.051 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 14 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3976179239,v1:127.0.0.1:6811/3976179239] [v2:127.0.0.1:6812/3976179239,v1:127.0.0.1:6813/3976179239] exists,up a6bbe05b-67fa-4e47-aa29-34c357800929 2024-09-16T08:21:53.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:21:53.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:21:53.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:21:53.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:21:53.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-16T08:21:53.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:21:53.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:21:53.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T08:21:53.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:21:53.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-16T08:21:53.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:21:53.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:21:53.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:21:53.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:21:53.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-16T08:21:53.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-16T08:21:53.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:21:53.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:21:53.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:21:53.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:21:53.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:21:53.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:21:53.057 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:21:53.057 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:21:53.057 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:21:53.057 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:21:53.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:21:53.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:21:53.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:21:53.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:21:53.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:21:53.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:21:53.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:21:53.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:21:53.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:21:53.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:21:53.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:21:53.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:21:53.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-16T08:21:53.058 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:21:53.060 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 7d4ed263-908a-42a3-9f79-b873eb24d702 2024-09-16T08:21:53.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7d4ed263-908a-42a3-9f79-b873eb24d702 2024-09-16T08:21:53.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 7d4ed263-908a-42a3-9f79-b873eb24d702' 2024-09-16T08:21:53.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:21:53.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCh6udmT+A8BRAALRC4QrgB5Di0IsDlLtEFeQ== 2024-09-16T08:21:53.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCh6udmT+A8BRAALRC4QrgB5Di0IsDlLtEFeQ=="}' 2024-09-16T08:21:53.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7d4ed263-908a-42a3-9f79-b873eb24d702 -i td/test-erasure-eio/2/new.json 2024-09-16T08:21:53.716 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:21:53.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-16T08:21:53.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCh6udmT+A8BRAALRC4QrgB5Di0IsDlLtEFeQ== --osd-uuid 7d4ed263-908a-42a3-9f79-b873eb24d702 2024-09-16T08:21:53.769 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:21:53.762+0000 7fca20de3880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:21:53.805 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:21:53.802+0000 7fca20de3880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:21:53.808 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:21:53.802+0000 7fca20de3880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:21:53.809 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:21:53.802+0000 7fca20de3880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:21:53.809 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:21:53.802+0000 7fca20de3880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:21:53.809 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:21:53.802+0000 7fca20de3880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:21:53.905 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:21:53.898+0000 7fca20de3880 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-16T08:21:59.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-16T08:21:59.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:22:00.001 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-16T08:22:00.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T08:22:00.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:22:00.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T08:22:00.576 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-16T08:22:00.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:22:00.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:22:00.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:22:00.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:22:00.612 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:22:00.606+0000 7f270de57880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:22:00.627 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:22:00.626+0000 7f270de57880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:22:00.629 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:22:00.626+0000 7f270de57880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:22:01.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T08:22:01.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:22:01.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T08:22:01.028 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:22:01.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:22:01.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:22:01.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:01.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:22:01.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:22:01.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:01.188 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:22:01.186+0000 7f270de57880 -1 Falling back to public interface 2024-09-16T08:22:01.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:02.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:02.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:02.491 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:22:02.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:22:02.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:22:02.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:02.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:03.778 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:22:03.778+0000 7f270de57880 -1 osd.2 0 log_to_monitors true 2024-09-16T08:22:03.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:03.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:03.906 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:22:03.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:22:03.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:03.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:22:04.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:05.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:05.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:05.482 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:22:05.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:22:05.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:05.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:22:05.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:06.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:06.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:06.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:22:06.964 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:22:06.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:06.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:22:07.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:08.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:08.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:08.583 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:22:08.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:22:08.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:08.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:22:09.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:10.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:10.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:10.155 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:22:10.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:22:10.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:10.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:22:10.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:11.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:11.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:11.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:22:11.757 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:22:11.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:11.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:22:12.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:13.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:13.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:13.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:22:13.182 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:22:13.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:13.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:22:13.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:14.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:14.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:14.749 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:22:14.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:22:14.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:14.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:22:15.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:16.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:16.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:16.300 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:22:16.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:22:16.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:16.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:22:17.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:18.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:18.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:18.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:22:18.169 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:22:18.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:18.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:22:18.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:19.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:19.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:19.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:22:19.800 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:22:19.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:19.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:22:21.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:22.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:22.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:22.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:22:22.036 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:22:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:22:22.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:23.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:23.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:23.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:22:23.755 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:22:23.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:23.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:22:24.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:25.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:25.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:25.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:22:25.954 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:22:25.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:25.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:22:26.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:27.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:27.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:27.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:22:27.699 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:22:27.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:27.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:22:28.234 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4238674375,v1:127.0.0.1:6819/4238674375] [v2:127.0.0.1:6820/4238674375,v1:127.0.0.1:6821/4238674375] exists,up 7d4ed263-908a-42a3-9f79-b873eb24d702 2024-09-16T08:22:28.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:22:28.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:22:28.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:22:28.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:22:28.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-16T08:22:28.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:22:28.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:22:28.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T08:22:28.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:22:28.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-16T08:22:28.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:22:28.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:22:28.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:22:28.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:22:28.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-16T08:22:28.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-16T08:22:28.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:22:28.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:22:28.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:22:28.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:22:28.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:22:28.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:22:28.239 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:22:28.239 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:22:28.239 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:22:28.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:22:28.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:22:28.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:22:28.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:22:28.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:22:28.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:22:28.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:22:28.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:22:28.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:22:28.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:22:28.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:22:28.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:22:28.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:22:28.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-16T08:22:28.243 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:22:28.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=236c36f2-4c38-4ba7-b03f-0612d961a8a7 2024-09-16T08:22:28.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 236c36f2-4c38-4ba7-b03f-0612d961a8a7' 2024-09-16T08:22:28.243 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 236c36f2-4c38-4ba7-b03f-0612d961a8a7 2024-09-16T08:22:28.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:22:28.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDE6udmvxXaDxAAQMqtCosWRkxfDTLXtDY/7g== 2024-09-16T08:22:28.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDE6udmvxXaDxAAQMqtCosWRkxfDTLXtDY/7g=="}' 2024-09-16T08:22:28.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 236c36f2-4c38-4ba7-b03f-0612d961a8a7 -i td/test-erasure-eio/3/new.json 2024-09-16T08:22:29.358 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:22:29.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-16T08:22:29.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDE6udmvxXaDxAAQMqtCosWRkxfDTLXtDY/7g== --osd-uuid 236c36f2-4c38-4ba7-b03f-0612d961a8a7 2024-09-16T08:22:29.414 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:22:29.410+0000 7f484e2b9880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:22:29.464 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:22:29.462+0000 7f484e2b9880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:22:29.466 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:22:29.466+0000 7f484e2b9880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:22:29.467 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:22:29.466+0000 7f484e2b9880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:22:29.467 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:22:29.466+0000 7f484e2b9880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:22:29.467 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:22:29.466+0000 7f484e2b9880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:22:29.555 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:22:29.554+0000 7f484e2b9880 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-16T08:22:35.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-16T08:22:35.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:22:35.745 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-16T08:22:35.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T08:22:35.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:22:36.309 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-16T08:22:36.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T08:22:36.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:22:36.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:22:36.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:22:36.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:22:36.345 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:22:36.341+0000 7efd9c3e9880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:22:36.347 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:22:36.345+0000 7efd9c3e9880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:22:36.349 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:22:36.345+0000 7efd9c3e9880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:22:36.644 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:22:36.641+0000 7efd9c3e9880 -1 Falling back to public interface 2024-09-16T08:22:36.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T08:22:36.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:22:36.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T08:22:36.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:22:36.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:22:36.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:36.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:22:36.693 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:22:36.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:36.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:22:37.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:38.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:38.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:38.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:22:38.144 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:22:38.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:38.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:22:38.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:39.439 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:22:39.437+0000 7efd9c3e9880 -1 osd.3 0 log_to_monitors true 2024-09-16T08:22:39.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:39.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:39.579 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:22:39.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:22:39.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:39.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:22:40.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:41.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:41.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:41.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:22:41.345 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:22:41.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:41.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:22:42.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:43.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:43.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:43.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:22:43.039 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:22:43.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:43.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:22:43.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:44.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:44.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:44.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:22:44.596 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:22:44.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:44.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:22:45.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:46.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:46.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:46.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:22:46.171 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:22:46.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:46.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:22:46.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:47.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:47.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:47.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:22:47.720 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:22:47.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:47.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:22:48.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:49.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:49.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:49.148 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:22:49.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:22:49.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:49.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:22:49.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:50.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:50.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:50.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:22:50.573 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:22:50.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:50.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:22:51.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:52.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:52.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:52.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:22:52.007 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:22:52.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:52.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:22:52.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:53.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:53.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:53.431 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:22:53.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:22:53.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:53.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:22:53.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:54.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:54.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:54.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:22:54.911 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:22:54.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:54.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:22:55.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:56.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:56.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:56.418 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:22:56.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:22:56.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:56.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:22:57.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:22:58.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:22:58.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:22:58.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:22:58.380 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:22:58.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:22:58.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:22:59.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:23:00.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:23:00.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:23:00.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:23:00.184 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:23:00.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:23:00.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:23:01.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:23:02.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:23:02.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:23:02.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:23:02.136 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:23:02.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:23:02.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:23:02.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:23:03.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:23:03.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:23:03.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:23:03.793 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:23:03.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:23:03.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:23:04.323 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1398662260,v1:127.0.0.1:6827/1398662260] [v2:127.0.0.1:6828/1398662260,v1:127.0.0.1:6829/1398662260] exists,up 236c36f2-4c38-4ba7-b03f-0612d961a8a7 2024-09-16T08:23:04.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:23:04.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:23:04.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:23:04.326 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-16T08:23:04.326 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T08:23:04.326 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T08:23:04.326 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:23:04.326 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:23:04.326 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:23:04.326 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.0.asok 2024-09-16T08:23:04.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-16T08:23:04.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.0.asok log flush 2024-09-16T08:23:04.421 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T08:23:04.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-16T08:23:04.436 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T08:20:51.800+0000 7fde01c09880 0 load: jerasure load: lrc 2024-09-16T08:23:04.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:270: TEST_rados_get_subread_eio_shard_1: local poolname=pool-jerasure 2024-09-16T08:23:04.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:271: TEST_rados_get_subread_eio_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2024-09-16T08:23:04.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T08:23:04.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-16T08:23:04.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-16T08:23:04.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-16T08:23:04.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-16T08:23:04.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-16T08:23:04.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-16T08:23:05.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-16T08:23:05.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-16T08:23:07.166 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-16T08:23:07.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T08:23:08.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-16T08:23:08.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:23:08.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:23:08.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:23:08.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:23:08.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:23:08.179 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:23:08.179 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:23:08.180 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:23:08.180 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:23:08.180 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:23:08.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:23:08.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:23:08.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:23:08.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:23:08.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:23:08.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:23:08.901 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:23:08.901 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:23:08.901 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:23:08.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:23:08.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:23:08.902 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:23:09.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803813 2024-09-16T08:23:09.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803813 2024-09-16T08:23:09.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803813' 2024-09-16T08:23:09.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:23:09.094 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:23:09.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=60129542173 2024-09-16T08:23:09.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 60129542173 2024-09-16T08:23:09.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803813 1-60129542173' 2024-09-16T08:23:09.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:23:09.257 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:23:09.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345948 2024-09-16T08:23:09.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345948 2024-09-16T08:23:09.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803813 1-60129542173 2-85899345948' 2024-09-16T08:23:09.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:23:09.407 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:23:09.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=133143986188 2024-09-16T08:23:09.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 133143986188 2024-09-16T08:23:09.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803813 1-60129542173 2-85899345948 3-133143986188' 2024-09-16T08:23:09.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:23:09.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803813 2024-09-16T08:23:09.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:23:09.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:23:09.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803813 2024-09-16T08:23:09.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:23:09.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803813 2024-09-16T08:23:09.567 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803813 2024-09-16T08:23:09.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803813' 2024-09-16T08:23:09.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:23:10.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803811 -lt 25769803813 2024-09-16T08:23:10.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T08:23:11.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T08:23:11.036 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:23:11.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803813 -lt 25769803813 2024-09-16T08:23:11.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:23:11.459 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-60129542173 2024-09-16T08:23:11.459 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:23:11.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:23:11.462 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-60129542173 2024-09-16T08:23:11.462 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:23:11.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=60129542173 2024-09-16T08:23:11.465 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 60129542173 2024-09-16T08:23:11.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 60129542173' 2024-09-16T08:23:11.466 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:23:11.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 60129542173 -lt 60129542173 2024-09-16T08:23:11.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:23:11.957 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-85899345948 2024-09-16T08:23:11.957 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:23:11.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:23:11.958 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-85899345948 2024-09-16T08:23:11.958 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:23:11.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345948 2024-09-16T08:23:11.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 85899345948' 2024-09-16T08:23:11.960 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 85899345948 2024-09-16T08:23:11.960 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:23:12.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345948 -lt 85899345948 2024-09-16T08:23:12.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:23:12.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-133143986188 2024-09-16T08:23:12.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:23:12.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:23:12.406 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-133143986188 2024-09-16T08:23:12.406 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:23:12.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=133143986188 2024-09-16T08:23:12.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 133143986188' 2024-09-16T08:23:12.408 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 133143986188 2024-09-16T08:23:12.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:23:12.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 133143986189 -lt 133143986188 2024-09-16T08:23:12.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:23:12.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:23:12.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:23:13.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:23:13.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:23:13.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:23:13.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:23:13.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:23:13.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:23:13.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:23:13.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:23:13.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:23:13.938 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:23:13.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:23:13.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:23:14.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:23:14.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:23:14.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:23:14.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:273: TEST_rados_get_subread_eio_shard_1: local shard_id=1 2024-09-16T08:23:14.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:274: TEST_rados_get_subread_eio_shard_1: rados_put_get_data eio td/test-erasure-eio 1 2024-09-16T08:23:14.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-09-16T08:23:14.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-16T08:23:14.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-16T08:23:14.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-16T08:23:14.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2024-09-16T08:23:14.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-16T08:23:14.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2024-09-16T08:23:14.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-16T08:23:14.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-79228-1 2024-09-16T08:23:14.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-79228-1 2024-09-16T08:23:14.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-16T08:23:14.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-16T08:23:14.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-79228-1 2024-09-16T08:23:14.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:23:14.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-16T08:23:14.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:23:14.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-16T08:23:14.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:23:14.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-16T08:23:14.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:23:14.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-16T08:23:14.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-79228-1 td/test-erasure-eio/ORIGINAL 2024-09-16T08:23:14.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-79228-1 td/test-erasure-eio 1 2024-09-16T08:23:14.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=ec 2024-09-16T08:23:14.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T08:23:14.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T08:23:14.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T08:23:14.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-jerasure 2024-09-16T08:23:14.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T08:23:14.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-eio-79228-1 2024-09-16T08:23:14.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T08:23:14.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/test-erasure-eio 2024-09-16T08:23:14.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T08:23:14.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=1 2024-09-16T08:23:14.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T08:23:14.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:23:14.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-jerasure obj-eio-79228-1 2024-09-16T08:23:14.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:23:14.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-79228-1 2024-09-16T08:23:14.684 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-79228-1 2024-09-16T08:23:14.684 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:23:15.156 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:23:15.156 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:23:15.156 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T08:23:15.156 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-16T08:23:15.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T08:23:15.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=1 2024-09-16T08:23:15.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' ec '!=' ec ']' 2024-09-16T08:23:15.157 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/test-erasure-eio/1/type 2024-09-16T08:23:15.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T08:23:15.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-16T08:23:15.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T08:23:15.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-16T08:23:15.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T08:23:15.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T08:23:15.160 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T08:23:15.160 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-16T08:23:15.160 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T08:23:15.161 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T08:23:15.162 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:23:15.162 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:23:15.162 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:23:15.162 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.1.asok 2024-09-16T08:23:15.162 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79228/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-16T08:23:15.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T08:23:15.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T08:23:15.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T08:23:15.266 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.1 2024-09-16T08:23:15.266 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T08:23:15.266 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T08:23:15.267 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:23:15.267 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:23:15.267 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:23:15.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.1.asok 2024-09-16T08:23:15.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T08:23:15.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-79228-1 1 2024-09-16T08:23:15.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-79228-1 2024-09-16T08:23:15.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T08:23:15.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T08:23:15.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-79228-1 2024-09-16T08:23:15.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-16T08:23:15.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-16T08:23:15.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-79228-1 td/test-erasure-eio/COPY 2024-09-16T08:23:15.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-16T08:23:15.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-16T08:23:15.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2024-09-16T08:23:15.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 1 + 1 2024-09-16T08:23:15.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: shard_id=2 2024-09-16T08:23:15.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-79228-1 td/test-erasure-eio 2 2024-09-16T08:23:15.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=ec 2024-09-16T08:23:15.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T08:23:15.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T08:23:15.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T08:23:15.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-jerasure 2024-09-16T08:23:15.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T08:23:15.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-eio-79228-1 2024-09-16T08:23:15.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T08:23:15.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/test-erasure-eio 2024-09-16T08:23:15.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T08:23:15.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=2 2024-09-16T08:23:15.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T08:23:15.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:23:15.418 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-jerasure obj-eio-79228-1 2024-09-16T08:23:15.418 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:23:15.418 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-79228-1 2024-09-16T08:23:15.418 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-79228-1 2024-09-16T08:23:15.418 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:23:15.858 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:23:15.858 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:23:15.858 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T08:23:15.859 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-16T08:23:15.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T08:23:15.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=2 2024-09-16T08:23:15.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' ec '!=' ec ']' 2024-09-16T08:23:15.860 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/test-erasure-eio/2/type 2024-09-16T08:23:15.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T08:23:15.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2024-09-16T08:23:15.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T08:23:15.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=2 2024-09-16T08:23:15.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T08:23:15.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T08:23:15.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T08:23:15.863 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.2 2024-09-16T08:23:15.863 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-16T08:23:15.864 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-16T08:23:15.864 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:23:15.864 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:23:15.864 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:23:15.865 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.2.asok 2024-09-16T08:23:15.865 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79228/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2024-09-16T08:23:15.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T08:23:15.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T08:23:15.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T08:23:15.957 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.2 2024-09-16T08:23:15.957 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-16T08:23:15.957 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-16T08:23:15.958 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:23:15.958 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:23:15.958 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:23:15.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.2.asok 2024-09-16T08:23:15.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T08:23:15.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.2.asok injectdataerr pool-jerasure obj-eio-79228-1 2 2024-09-16T08:23:16.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-79228-1 fail 2024-09-16T08:23:16.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T08:23:16.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T08:23:16.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-79228-1 2024-09-16T08:23:16.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-16T08:23:16.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-16T08:23:16.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-eio-79228-1 td/test-erasure-eio/COPY 2024-09-16T08:23:16.104 INFO:tasks.workunit.client.0.smithi162.stderr:error getting pool-jerasure/obj-eio-79228-1: (5) Input/output error 2024-09-16T08:23:16.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-16T08:23:16.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:184: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2024-09-16T08:23:16.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:275: TEST_rados_get_subread_eio_shard_1: delete_erasure_coded_pool pool-jerasure 2024-09-16T08:23:16.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T08:23:16.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-16T08:23:16.695 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-16T08:23:16.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-16T08:23:17.514 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-16T08:23:17.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-16T08:23:17.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T08:23:17.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T08:23:17.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T08:23:17.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:23:17.526 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:23:17.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:23:17.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:23:17.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:23:17.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:23:17.842 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T08:23:17.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T08:23:17.844 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T08:23:17.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T08:23:17.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T08:23:17.847 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T08:23:17.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:23:17.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T08:23:17.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T08:23:17.851 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:23:17.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:23:17.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T08:23:17.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T08:23:17.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T08:23:17.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:23:17.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:23:17.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T08:23:17.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T08:23:17.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T08:23:17.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-16T08:23:17.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-16T08:23:17.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-16T08:23:17.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-16T08:23:17.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T08:23:17.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T08:23:17.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T08:23:17.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:23:17.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:23:17.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:23:17.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:23:17.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:23:17.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:23:17.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T08:23:17.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T08:23:17.891 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T08:23:17.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T08:23:17.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T08:23:17.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T08:23:17.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:23:17.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T08:23:17.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T08:23:17.896 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:23:17.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:23:17.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T08:23:17.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T08:23:17.902 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T08:23:17.902 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:23:17.902 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:23:17.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T08:23:17.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T08:23:17.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T08:23:17.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-16T08:23:17.906 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T08:23:17.906 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:23:17.906 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:23:17.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79228 2024-09-16T08:23:17.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T08:23:17.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T08:23:17.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T08:23:17.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-16T08:23:17.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-16T08:23:17.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-16T08:23:17.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T08:23:17.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T08:23:17.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T08:23:17.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-16T08:23:17.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-16T08:23:18.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T08:23:18.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:23:18.332 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:23:18.332 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:23:18.332 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:23:18.332 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:23:18.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:23:18.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T08:23:18.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T08:23:18.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T08:23:18.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T08:23:18.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T08:23:18.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T08:23:18.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T08:23:18.571 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T08:23:18.571 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T08:23:18.571 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T08:23:18.572 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:23:18.572 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:23:18.572 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:23:18.572 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T08:23:18.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T08:23:18.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get fsid 2024-09-16T08:23:18.674 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T08:23:18.674 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T08:23:18.674 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T08:23:18.674 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T08:23:18.674 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T08:23:18.675 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T08:23:18.675 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T08:23:18.675 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T08:23:18.676 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:23:18.676 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:23:18.676 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:23:18.676 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T08:23:18.676 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T08:23:18.676 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get mon_host 2024-09-16T08:23:18.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-16T08:23:18.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-16T08:23:18.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T08:23:18.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T08:23:18.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T08:23:18.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-16T08:23:18.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T08:23:19.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T08:23:19.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:23:19.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:23:19.684 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:23:19.684 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:23:19.684 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:23:19.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:23:19.685 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T08:23:19.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T08:23:19.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-16T08:23:19.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T08:23:20.075 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-16T08:23:20.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T08:23:21.087 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-16T08:23:21.087 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T08:23:21.087 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T08:23:21.088 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:23:21.088 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:23:21.088 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:23:21.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T08:23:21.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-16T08:23:21.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-mon.a.asok log flush 2024-09-16T08:23:21.170 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T08:23:21.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-16T08:23:21.182 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T08:23:18.381+0000 7ff47b768d40 0 load: jerasure load: lrc 2024-09-16T08:23:21.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_missing td/test-erasure-eio 2024-09-16T08:23:21.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:282: TEST_rados_get_subread_missing: local dir=td/test-erasure-eio 2024-09-16T08:23:21.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:283: TEST_rados_get_subread_missing: setup_osds 4 2024-09-16T08:23:21.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-16T08:23:21.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-16T08:23:21.184 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-16T08:23:21.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-16T08:23:21.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:23:21.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-16T08:23:21.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:23:21.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:23:21.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T08:23:21.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:23:21.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-16T08:23:21.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:23:21.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:23:21.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:23:21.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:23:21.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-16T08:23:21.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-16T08:23:21.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:23:21.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:23:21.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:23:21.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:23:21.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:23:21.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:23:21.190 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:23:21.190 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:23:21.191 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:23:21.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:23:21.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:23:21.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:23:21.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:23:21.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:23:21.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:23:21.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:23:21.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:23:21.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:23:21.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:23:21.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:23:21.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:23:21.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:23:21.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-16T08:23:21.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:23:21.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b3ab8007-86d6-4b50-8261-2c34efb4b1cd 2024-09-16T08:23:21.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b3ab8007-86d6-4b50-8261-2c34efb4b1cd' 2024-09-16T08:23:21.193 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 b3ab8007-86d6-4b50-8261-2c34efb4b1cd 2024-09-16T08:23:21.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:23:21.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD56udmQnP/DBAARUxLo5ZDLyNQxO4xfFDEvA== 2024-09-16T08:23:21.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD56udmQnP/DBAARUxLo5ZDLyNQxO4xfFDEvA=="}' 2024-09-16T08:23:21.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b3ab8007-86d6-4b50-8261-2c34efb4b1cd -i td/test-erasure-eio/0/new.json 2024-09-16T08:23:21.575 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:23:21.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-16T08:23:21.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD56udmQnP/DBAARUxLo5ZDLyNQxO4xfFDEvA== --osd-uuid b3ab8007-86d6-4b50-8261-2c34efb4b1cd 2024-09-16T08:23:21.622 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:23:21.620+0000 7f46c3d3e880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:23:21.626 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:23:21.624+0000 7f46c3d3e880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:23:21.628 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:23:21.624+0000 7f46c3d3e880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:23:21.629 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:23:21.624+0000 7f46c3d3e880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T08:23:21.629 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:23:21.624+0000 7f46c3d3e880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T08:23:21.629 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:23:21.624+0000 7f46c3d3e880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T08:23:21.706 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:23:21.704+0000 7f46c3d3e880 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-16T08:23:27.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-16T08:23:27.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:23:27.409 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T08:23:27.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T08:23:27.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:23:27.966 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T08:23:27.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T08:23:27.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:23:27.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:23:27.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:23:27.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:23:27.993 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:23:27.988+0000 7f9d0adda880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:23:28.021 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:23:28.016+0000 7f9d0adda880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:23:28.023 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:23:28.020+0000 7f9d0adda880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:23:28.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T08:23:28.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:23:28.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T08:23:28.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:23:28.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:23:28.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:23:28.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:23:28.396 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:23:28.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:23:28.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:23:28.576 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:23:28.572+0000 7f9d0adda880 -1 Falling back to public interface 2024-09-16T08:23:28.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:23:29.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:23:29.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:23:29.850 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:23:29.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:23:29.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:23:29.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:23:30.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:23:31.205 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:23:31.200+0000 7f9d0adda880 -1 osd.0 0 log_to_monitors true 2024-09-16T08:23:31.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:23:31.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:23:31.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:23:31.287 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:23:31.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:23:31.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:23:31.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:23:32.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:23:32.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:23:32.877 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:23:32.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:23:32.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:23:32.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:23:33.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:23:34.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:23:34.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:23:34.309 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:23:34.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:23:34.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:23:34.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:23:34.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:23:35.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:23:35.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:23:35.739 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:23:35.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:23:35.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:23:35.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:23:36.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:23:37.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:23:37.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:23:37.205 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:23:37.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:23:37.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:23:37.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:23:37.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:23:38.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:23:38.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:23:38.636 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:23:38.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:23:38.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:23:38.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:23:39.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:23:40.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:23:40.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:23:40.066 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:23:40.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:23:40.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:23:40.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:23:40.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:23:41.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:23:41.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:23:41.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:23:41.516 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:23:41.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:23:41.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:23:42.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:23:43.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:23:43.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:23:43.016 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:23:43.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:23:43.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:23:43.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:23:43.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:23:44.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:23:44.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:23:44.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:23:44.450 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:23:44.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:23:44.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:23:45.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:23:46.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:23:46.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:23:46.199 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:23:46.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:23:46.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:23:46.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:23:47.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:23:48.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:23:48.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:23:48.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:23:48.149 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:23:48.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:23:48.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:23:48.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:23:49.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:23:49.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:23:49.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:23:49.643 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:23:49.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:23:49.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:23:50.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:23:51.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:23:51.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:23:51.167 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:23:51.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:23:51.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:23:51.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:23:51.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:23:52.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:23:52.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:23:52.685 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:23:52.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:23:52.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:23:52.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:23:53.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:23:54.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:23:54.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:23:54.292 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:23:54.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:23:54.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:23:54.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:23:54.930 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3792585413,v1:127.0.0.1:6803/3792585413] [v2:127.0.0.1:6804/3792585413,v1:127.0.0.1:6805/3792585413] exists,up b3ab8007-86d6-4b50-8261-2c34efb4b1cd 2024-09-16T08:23:54.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:23:54.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:23:54.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:23:54.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:23:54.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-16T08:23:54.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:23:54.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:23:54.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T08:23:54.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:23:54.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-16T08:23:54.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:23:54.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:23:54.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:23:54.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:23:54.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-16T08:23:54.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-16T08:23:54.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:23:54.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:23:54.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:23:54.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:23:54.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:23:54.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:23:54.935 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:23:54.935 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:23:54.935 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:23:54.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:23:54.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:23:54.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:23:54.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:23:54.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:23:54.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:23:54.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:23:54.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:23:54.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:23:54.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:23:54.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:23:54.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:23:54.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:23:54.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-16T08:23:54.938 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:23:54.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7e674770-121c-44e3-9487-9dad2984bac0 2024-09-16T08:23:54.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7e674770-121c-44e3-9487-9dad2984bac0' 2024-09-16T08:23:54.939 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 7e674770-121c-44e3-9487-9dad2984bac0 2024-09-16T08:23:54.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:23:54.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAa6+dmBRRxORAAOJtDTULg060C0VXpd2J3EA== 2024-09-16T08:23:54.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAa6+dmBRRxORAAOJtDTULg060C0VXpd2J3EA=="}' 2024-09-16T08:23:54.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7e674770-121c-44e3-9487-9dad2984bac0 -i td/test-erasure-eio/1/new.json 2024-09-16T08:23:55.679 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:23:55.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-16T08:23:55.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAa6+dmBRRxORAAOJtDTULg060C0VXpd2J3EA== --osd-uuid 7e674770-121c-44e3-9487-9dad2984bac0 2024-09-16T08:23:55.736 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:23:55.732+0000 7fc41bcbf880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:23:55.769 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:23:55.764+0000 7fc41bcbf880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:23:55.771 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:23:55.768+0000 7fc41bcbf880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:23:55.772 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:23:55.768+0000 7fc41bcbf880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T08:23:55.772 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:23:55.768+0000 7fc41bcbf880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T08:23:55.772 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:23:55.768+0000 7fc41bcbf880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T08:23:55.869 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:23:55.864+0000 7fc41bcbf880 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-16T08:24:01.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-16T08:24:01.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:24:01.725 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-16T08:24:01.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T08:24:01.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:24:02.336 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-16T08:24:02.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T08:24:02.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:24:02.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:24:02.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:24:02.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:24:02.370 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:24:02.368+0000 7fd9581ac880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:24:02.391 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:24:02.388+0000 7fd9581ac880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:24:02.393 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:24:02.388+0000 7fd9581ac880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:24:02.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T08:24:02.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:24:02.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T08:24:02.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:24:02.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:24:02.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:02.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:24:02.797 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:24:02.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:02.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:24:02.988 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:24:02.984+0000 7fd9581ac880 -1 Falling back to public interface 2024-09-16T08:24:03.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:04.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:04.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:04.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:24:04.291 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:24:04.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:04.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:24:04.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:05.454 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:24:05.451+0000 7fd9581ac880 -1 osd.1 0 log_to_monitors true 2024-09-16T08:24:05.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:05.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:05.761 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:24:05.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:24:05.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:05.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:24:06.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:07.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:07.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:07.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:24:07.241 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:24:07.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:07.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:24:07.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:08.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:08.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:08.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:24:08.672 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:24:08.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:08.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:24:09.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:10.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:10.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:10.262 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:24:10.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:24:10.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:10.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:24:10.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:11.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:11.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:11.702 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:24:11.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:24:11.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:11.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:24:12.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:13.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:13.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:13.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:24:13.223 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:24:13.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:13.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:24:13.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:14.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:14.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:14.768 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:24:14.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:24:14.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:14.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:24:15.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:16.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:16.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:16.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:24:16.316 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:24:16.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:16.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:24:16.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:17.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:17.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:17.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:24:17.736 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:24:17.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:17.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:24:18.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:19.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:19.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:19.578 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:24:19.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:24:19.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:19.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:24:20.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:21.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:21.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:21.105 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:24:21.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:24:21.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:21.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:24:21.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:22.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:22.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:22.937 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:24:22.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:24:22.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:22.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:24:23.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:24.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:24.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:24.723 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:24:24.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:24:24.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:24.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:24:25.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:26.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:26.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:26.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:24:26.609 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:24:26.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:26.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:24:27.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:28.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:28.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:28.377 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:24:28.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:24:28.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:28.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:24:28.942 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2146214320,v1:127.0.0.1:6811/2146214320] [v2:127.0.0.1:6812/2146214320,v1:127.0.0.1:6813/2146214320] exists,up 7e674770-121c-44e3-9487-9dad2984bac0 2024-09-16T08:24:28.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:24:28.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:24:28.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:24:28.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:24:28.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-16T08:24:28.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:24:28.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:24:28.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T08:24:28.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:24:28.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-16T08:24:28.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:24:28.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:24:28.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:24:28.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:24:28.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-16T08:24:28.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-16T08:24:28.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:24:28.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:24:28.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:24:28.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:24:28.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:24:28.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:24:28.947 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:24:28.947 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:24:28.948 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:24:28.949 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:24:28.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:24:28.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:24:28.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:24:28.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:24:28.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:24:28.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:24:28.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:24:28.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:24:28.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:24:28.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:24:28.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:24:28.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:24:28.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-16T08:24:28.950 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:24:28.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1c5ce5de-ba81-4b6a-92a7-396142aeb579 2024-09-16T08:24:28.951 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 1c5ce5de-ba81-4b6a-92a7-396142aeb579 2024-09-16T08:24:28.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 1c5ce5de-ba81-4b6a-92a7-396142aeb579' 2024-09-16T08:24:28.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:24:28.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA86+dmTt9BOhAAHvbnOZNG7BgF+A1KznrAjQ== 2024-09-16T08:24:28.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA86+dmTt9BOhAAHvbnOZNG7BgF+A1KznrAjQ=="}' 2024-09-16T08:24:28.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1c5ce5de-ba81-4b6a-92a7-396142aeb579 -i td/test-erasure-eio/2/new.json 2024-09-16T08:24:29.741 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:24:29.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-16T08:24:29.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA86+dmTt9BOhAAHvbnOZNG7BgF+A1KznrAjQ== --osd-uuid 1c5ce5de-ba81-4b6a-92a7-396142aeb579 2024-09-16T08:24:29.798 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:24:29.791+0000 7f7f870aa880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:24:29.800 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:24:29.795+0000 7f7f870aa880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:24:29.802 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:24:29.799+0000 7f7f870aa880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:24:29.803 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:24:29.799+0000 7f7f870aa880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:24:29.803 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:24:29.799+0000 7f7f870aa880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:24:29.803 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:24:29.799+0000 7f7f870aa880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:24:29.872 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:24:29.867+0000 7f7f870aa880 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-16T08:24:35.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-16T08:24:35.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:24:35.729 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-16T08:24:35.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T08:24:35.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:24:36.330 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-16T08:24:36.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T08:24:36.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:24:36.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:24:36.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:24:36.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:24:36.364 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:24:36.359+0000 7efeae273880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:24:36.388 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:24:36.383+0000 7efeae273880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:24:36.389 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:24:36.383+0000 7efeae273880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:24:36.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T08:24:36.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:24:36.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T08:24:36.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:24:36.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:24:36.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:36.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:24:36.759 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:24:36.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:36.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:24:36.984 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:24:36.979+0000 7efeae273880 -1 Falling back to public interface 2024-09-16T08:24:37.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:38.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:38.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:38.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:24:38.182 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:24:38.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:38.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:24:38.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:39.566 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:24:39.563+0000 7efeae273880 -1 osd.2 0 log_to_monitors true 2024-09-16T08:24:39.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:39.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:39.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:24:39.601 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:24:39.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:39.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:24:40.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:41.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:41.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:41.291 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:24:41.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:24:41.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:41.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:24:41.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:42.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:42.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:42.823 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:24:42.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:24:42.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:42.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:24:43.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:44.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:44.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:44.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:24:44.509 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:24:44.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:44.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:24:44.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:45.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:45.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:45.945 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:24:45.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:24:45.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:45.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:24:46.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:47.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:47.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:47.395 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:24:47.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:24:47.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:47.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:24:47.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:48.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:48.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:48.831 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:24:48.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:24:48.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:48.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:24:49.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:50.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:50.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:50.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:24:50.227 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:24:50.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:50.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:24:50.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:51.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:51.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:51.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:24:51.790 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:24:51.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:51.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:24:52.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:53.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:53.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:53.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:24:53.412 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:24:53.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:53.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:24:54.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:55.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:55.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:55.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:24:55.427 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:24:55.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:55.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:24:56.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:57.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:57.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:57.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:24:57.265 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:24:57.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:57.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:24:58.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:24:59.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:24:59.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:24:59.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:24:59.004 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:24:59.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:24:59.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:24:59.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:25:00.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:25:00.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:25:00.599 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:25:00.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:25:00.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:25:00.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:25:01.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:25:02.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:25:02.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:25:02.751 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:25:02.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:25:02.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:25:02.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:25:03.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:25:04.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:25:04.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:25:04.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:25:04.194 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:25:04.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:25:04.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:25:04.865 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2961215075,v1:127.0.0.1:6819/2961215075] [v2:127.0.0.1:6820/2961215075,v1:127.0.0.1:6821/2961215075] exists,up 1c5ce5de-ba81-4b6a-92a7-396142aeb579 2024-09-16T08:25:04.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:25:04.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:25:04.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:25:04.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:25:04.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-16T08:25:04.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:25:04.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:25:04.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T08:25:04.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:25:04.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-16T08:25:04.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:25:04.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:25:04.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:25:04.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:25:04.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-16T08:25:04.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-16T08:25:04.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:25:04.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:25:04.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:25:04.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:25:04.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:25:04.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:25:04.870 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:25:04.870 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:25:04.870 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:25:04.872 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:25:04.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:25:04.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:25:04.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:25:04.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:25:04.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:25:04.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:25:04.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:25:04.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:25:04.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:25:04.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:25:04.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:25:04.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:25:04.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-16T08:25:04.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:25:04.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=039ef998-3c0b-4d6a-831c-f5ec5deed981 2024-09-16T08:25:04.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 039ef998-3c0b-4d6a-831c-f5ec5deed981' 2024-09-16T08:25:04.874 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 039ef998-3c0b-4d6a-831c-f5ec5deed981 2024-09-16T08:25:04.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:25:04.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBg6+dmduyZNRAAk9UZuRlDCn889qgZn5Xp/w== 2024-09-16T08:25:04.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBg6+dmduyZNRAAk9UZuRlDCn889qgZn5Xp/w=="}' 2024-09-16T08:25:04.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 039ef998-3c0b-4d6a-831c-f5ec5deed981 -i td/test-erasure-eio/3/new.json 2024-09-16T08:25:05.446 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:25:05.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-16T08:25:05.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBg6+dmduyZNRAAk9UZuRlDCn889qgZn5Xp/w== --osd-uuid 039ef998-3c0b-4d6a-831c-f5ec5deed981 2024-09-16T08:25:05.498 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:25:05.494+0000 7f5240755880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:25:05.513 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:25:05.510+0000 7f5240755880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:25:05.515 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:25:05.514+0000 7f5240755880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:25:05.516 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:25:05.514+0000 7f5240755880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:25:05.516 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:25:05.514+0000 7f5240755880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:25:05.516 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:25:05.514+0000 7f5240755880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:25:05.560 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:25:05.558+0000 7f5240755880 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-16T08:25:11.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-16T08:25:11.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:25:11.629 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-16T08:25:11.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T08:25:11.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:25:12.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T08:25:12.198 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-16T08:25:12.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:25:12.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:25:12.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:25:12.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:25:12.232 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:25:12.230+0000 7fdcc56fe880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:25:12.235 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:25:12.234+0000 7fdcc56fe880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:25:12.237 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:25:12.234+0000 7fdcc56fe880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:25:12.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T08:25:12.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:25:12.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T08:25:12.715 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:25:12.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:25:12.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:25:12.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:25:12.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:25:12.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:25:12.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:25:12.800 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:25:12.798+0000 7fdcc56fe880 -1 Falling back to public interface 2024-09-16T08:25:13.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:25:14.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:25:14.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:25:14.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:25:14.220 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:25:14.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:25:14.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:25:14.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:25:15.380 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:25:15.378+0000 7fdcc56fe880 -1 osd.3 0 log_to_monitors true 2024-09-16T08:25:15.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:25:15.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:25:15.686 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:25:15.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:25:15.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:25:15.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:25:16.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:25:17.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:25:17.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:25:17.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:25:17.506 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:25:17.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:25:17.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:25:18.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:25:19.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:25:19.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:25:19.194 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:25:19.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:25:19.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:25:19.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:25:19.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:25:20.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:25:20.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:25:20.650 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:25:20.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:25:20.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:25:20.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:25:21.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:25:22.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:25:22.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:25:22.243 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:25:22.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:25:22.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:25:22.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:25:22.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:25:23.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:25:23.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:25:23.712 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:25:23.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:25:23.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:25:23.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:25:24.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:25:25.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:25:25.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:25:25.273 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:25:25.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:25:25.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:25:25.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:25:25.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:25:26.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:25:26.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:25:26.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:25:26.702 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:25:26.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:25:26.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:25:27.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:25:28.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:25:28.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:25:28.231 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:25:28.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:25:28.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:25:28.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:25:28.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:25:29.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:25:29.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:25:29.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:25:29.808 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:25:29.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:25:29.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:25:30.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:25:31.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:25:31.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:25:31.504 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:25:31.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:25:31.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:25:31.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:25:32.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:25:33.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:25:33.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:25:33.103 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:25:33.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:25:33.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:25:33.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:25:34.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:25:35.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:25:35.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:25:35.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:25:35.247 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:25:35.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:25:35.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:25:36.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:25:37.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:25:37.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:25:37.270 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:25:37.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:25:37.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:25:37.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:25:37.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:25:38.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:25:38.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:25:38.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:25:38.989 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:25:38.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:25:38.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:25:39.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:25:40.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:25:40.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:25:40.668 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:25:40.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:25:40.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:25:40.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:25:41.339 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2128323932,v1:127.0.0.1:6827/2128323932] [v2:127.0.0.1:6828/2128323932,v1:127.0.0.1:6829/2128323932] exists,up 039ef998-3c0b-4d6a-831c-f5ec5deed981 2024-09-16T08:25:41.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:25:41.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:25:41.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:25:41.341 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-16T08:25:41.341 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T08:25:41.341 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T08:25:41.342 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:25:41.342 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:25:41.342 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:25:41.342 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.0.asok 2024-09-16T08:25:41.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-16T08:25:41.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.0.asok log flush 2024-09-16T08:25:41.434 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T08:25:41.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-16T08:25:41.450 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T08:23:28.588+0000 7f9d0adda880 0 load: jerasure load: lrc 2024-09-16T08:25:41.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:285: TEST_rados_get_subread_missing: local poolname=pool-jerasure 2024-09-16T08:25:41.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:286: TEST_rados_get_subread_missing: create_erasure_coded_pool pool-jerasure 2 1 2024-09-16T08:25:41.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T08:25:41.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-16T08:25:41.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-16T08:25:41.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-16T08:25:41.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-16T08:25:41.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-16T08:25:41.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-16T08:25:42.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-16T08:25:42.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-16T08:25:43.459 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-16T08:25:43.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T08:25:44.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-16T08:25:44.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:25:44.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:25:44.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:25:44.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:25:44.477 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:25:44.477 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:25:44.477 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:25:44.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:25:44.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:25:44.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:25:44.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:25:44.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:25:44.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:25:44.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:25:44.703 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:25:45.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:25:45.129 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:25:45.130 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:25:45.130 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:25:45.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:25:45.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:25:45.130 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:25:45.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803813 2024-09-16T08:25:45.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803813 2024-09-16T08:25:45.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803813' 2024-09-16T08:25:45.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:25:45.308 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:25:45.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=55834574883 2024-09-16T08:25:45.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 55834574883 2024-09-16T08:25:45.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803813 1-55834574883' 2024-09-16T08:25:45.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:25:45.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:25:45.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378647 2024-09-16T08:25:45.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378647 2024-09-16T08:25:45.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803813 1-55834574883 2-81604378647' 2024-09-16T08:25:45.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:25:45.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:25:45.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=124554051601 2024-09-16T08:25:45.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 124554051601 2024-09-16T08:25:45.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803813 1-55834574883 2-81604378647 3-124554051601' 2024-09-16T08:25:45.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:25:45.783 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803813 2024-09-16T08:25:45.783 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:25:45.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:25:45.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803813 2024-09-16T08:25:45.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:25:45.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803813 2024-09-16T08:25:45.787 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803813 2024-09-16T08:25:45.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803813' 2024-09-16T08:25:45.788 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:25:46.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803811 -lt 25769803813 2024-09-16T08:25:46.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T08:25:47.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T08:25:47.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:25:47.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803813 -lt 25769803813 2024-09-16T08:25:47.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:25:47.694 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-55834574883 2024-09-16T08:25:47.694 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:25:47.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:25:47.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-55834574883 2024-09-16T08:25:47.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:25:47.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=55834574883 2024-09-16T08:25:47.699 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574883 2024-09-16T08:25:47.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 55834574883' 2024-09-16T08:25:47.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:25:48.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 55834574883 -lt 55834574883 2024-09-16T08:25:48.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:25:48.151 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378647 2024-09-16T08:25:48.151 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:25:48.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:25:48.154 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378647 2024-09-16T08:25:48.155 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:25:48.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378647 2024-09-16T08:25:48.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378647' 2024-09-16T08:25:48.157 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 81604378647 2024-09-16T08:25:48.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:25:48.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378647 -lt 81604378647 2024-09-16T08:25:48.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:25:48.572 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-124554051601 2024-09-16T08:25:48.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:25:48.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:25:48.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-124554051601 2024-09-16T08:25:48.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:25:48.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=124554051601 2024-09-16T08:25:48.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 124554051601' 2024-09-16T08:25:48.579 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 124554051601 2024-09-16T08:25:48.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:25:48.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 124554051602 -lt 124554051601 2024-09-16T08:25:48.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:25:48.997 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:25:48.997 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:25:49.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:25:49.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:25:49.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:25:49.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:25:49.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:25:49.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:25:49.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:25:49.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:25:49.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:25:49.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:25:49.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:25:49.997 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:25:50.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:25:50.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:25:50.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:25:50.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:288: TEST_rados_get_subread_missing: local shard_id=1 2024-09-16T08:25:50.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:289: TEST_rados_get_subread_missing: rados_put_get_data remove td/test-erasure-eio 1 2024-09-16T08:25:50.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=remove 2024-09-16T08:25:50.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-16T08:25:50.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-16T08:25:50.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-16T08:25:50.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2024-09-16T08:25:50.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-16T08:25:50.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2024-09-16T08:25:50.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-16T08:25:50.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-remove-79228-1 2024-09-16T08:25:50.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-remove-79228-1 2024-09-16T08:25:50.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-16T08:25:50.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-16T08:25:50.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-remove-79228-1 2024-09-16T08:25:50.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:25:50.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-16T08:25:50.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:25:50.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-16T08:25:50.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:25:50.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-16T08:25:50.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:25:50.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-16T08:25:50.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-remove-79228-1 td/test-erasure-eio/ORIGINAL 2024-09-16T08:25:50.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_remove ec data pool-jerasure obj-remove-79228-1 td/test-erasure-eio 1 2024-09-16T08:25:50.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:127: inject_remove: local pooltype=ec 2024-09-16T08:25:50.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:128: inject_remove: shift 2024-09-16T08:25:50.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:129: inject_remove: local which=data 2024-09-16T08:25:50.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:130: inject_remove: shift 2024-09-16T08:25:50.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:131: inject_remove: local poolname=pool-jerasure 2024-09-16T08:25:50.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:132: inject_remove: shift 2024-09-16T08:25:50.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:133: inject_remove: local objname=obj-remove-79228-1 2024-09-16T08:25:50.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:134: inject_remove: shift 2024-09-16T08:25:50.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:135: inject_remove: local dir=td/test-erasure-eio 2024-09-16T08:25:50.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:136: inject_remove: shift 2024-09-16T08:25:50.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:137: inject_remove: local shard_id=1 2024-09-16T08:25:50.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:138: inject_remove: shift 2024-09-16T08:25:50.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:25:50.741 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: get_osds pool-jerasure obj-remove-79228-1 2024-09-16T08:25:50.741 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:25:50.741 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-remove-79228-1 2024-09-16T08:25:50.741 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-remove-79228-1 2024-09-16T08:25:50.742 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:25:51.145 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:25:51.145 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:25:51.146 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T08:25:51.146 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-16T08:25:51.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: local -a initial_osds 2024-09-16T08:25:51.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:141: inject_remove: local osd_id=1 2024-09-16T08:25:51.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:142: inject_remove: objectstore_tool td/test-erasure-eio 1 obj-remove-79228-1 remove 2024-09-16T08:25:51.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-16T08:25:51.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T08:25:51.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-16T08:25:51.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T08:25:51.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-remove-79228-1 remove 2024-09-16T08:25:51.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-16T08:25:51.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T08:25:51.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-16T08:25:51.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T08:25:51.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-16T08:25:51.149 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:25:51.149 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:25:51.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:25:51.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:25:51.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:25:52.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:25:52.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-remove-79228-1 remove 2024-09-16T08:25:52.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-16T08:25:52.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T08:25:52.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-16T08:25:52.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T08:25:52.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-16T08:25:52.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-remove-79228-1 remove 2024-09-16T08:25:53.855 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#2:bc16985e:::obj-remove-79228-1:head# 2024-09-16T08:25:54.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-16T08:25:54.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-16T08:25:54.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T08:25:54.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T08:25:54.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T08:25:54.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-16T08:25:54.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:25:54.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:25:54.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:25:54.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:25:54.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-16T08:25:54.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-16T08:25:54.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T08:25:54.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T08:25:54.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:25:54.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T08:25:54.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:25:54.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:25:54.415 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:25:54.415 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:25:54.415 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:25:54.415 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:25:54.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:25:54.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:25:54.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:25:54.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:25:54.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:25:54.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:25:54.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:25:54.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:25:54.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T08:25:54.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T08:25:54.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-16T08:25:54.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T08:25:54.417 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-09-16T08:25:54.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:25:54.418 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-16T08:25:54.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T08:25:54.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T08:25:54.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T08:25:54.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T08:25:54.452 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:25:54.449+0000 7f6005915880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:25:54.456 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:25:54.453+0000 7f6005915880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:25:54.458 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:25:54.457+0000 7f6005915880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:25:54.749 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:25:54.745+0000 7f6005915880 -1 Falling back to public interface 2024-09-16T08:25:54.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T08:25:54.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:25:54.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T08:25:54.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:25:54.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:25:54.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:25:54.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:25:54.850 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T08:25:54.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:25:54.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:25:55.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:25:56.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:25:56.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:25:56.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:25:56.381 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:25:56.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:25:56.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:25:56.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:25:56.899 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:25:56.897+0000 7f6005915880 -1 osd.1 38 log_to_monitors true 2024-09-16T08:25:57.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:25:57.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:25:57.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:25:57.799 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:25:57.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:25:57.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:25:58.539 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 39 last_clean_interval [13,38) [v2:127.0.0.1:6810/1653665761,v1:127.0.0.1:6811/1653665761] [v2:127.0.0.1:6812/1653665761,v1:127.0.0.1:6813/1653665761] exists,up 7e674770-121c-44e3-9487-9dad2984bac0 2024-09-16T08:25:58.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:25:58.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:25:58.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:25:58.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-16T08:25:58.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:25:58.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:25:58.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:25:58.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:25:58.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:25:58.541 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:25:58.541 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:25:58.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:25:58.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:25:58.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:25:58.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:25:58.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:25:58.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:25:58.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:25:58.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:25:59.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:25:59.245 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:25:59.245 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:25:59.245 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:25:59.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:25:59.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:25:59.246 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:25:59.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803817 2024-09-16T08:25:59.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803817 2024-09-16T08:25:59.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803817' 2024-09-16T08:25:59.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:25:59.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:25:59.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=184683593731 2024-09-16T08:25:59.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 184683593731 2024-09-16T08:25:59.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803817 1-184683593731' 2024-09-16T08:25:59.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:25:59.584 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:25:59.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378652 2024-09-16T08:25:59.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378652 2024-09-16T08:25:59.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803817 1-184683593731 2-81604378652' 2024-09-16T08:25:59.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:25:59.734 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:25:59.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=124554051606 2024-09-16T08:25:59.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 124554051606 2024-09-16T08:25:59.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803817 1-184683593731 2-81604378652 3-124554051606' 2024-09-16T08:25:59.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:25:59.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803817 2024-09-16T08:25:59.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:25:59.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:25:59.911 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803817 2024-09-16T08:25:59.911 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:25:59.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803817 2024-09-16T08:25:59.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803817' 2024-09-16T08:25:59.914 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803817 2024-09-16T08:25:59.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:26:00.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803815 -lt 25769803817 2024-09-16T08:26:00.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T08:26:01.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T08:26:01.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:26:01.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803818 -lt 25769803817 2024-09-16T08:26:01.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:26:01.848 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-184683593731 2024-09-16T08:26:01.849 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:26:01.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:26:01.852 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-184683593731 2024-09-16T08:26:01.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:26:01.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=184683593731 2024-09-16T08:26:01.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 184683593731' 2024-09-16T08:26:01.855 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 184683593731 2024-09-16T08:26:01.856 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:26:02.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 184683593731 -lt 184683593731 2024-09-16T08:26:02.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:26:02.297 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378652 2024-09-16T08:26:02.297 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:26:02.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:26:02.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378652 2024-09-16T08:26:02.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:26:02.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378652 2024-09-16T08:26:02.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378652' 2024-09-16T08:26:02.303 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 81604378652 2024-09-16T08:26:02.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:26:02.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378652 -lt 81604378652 2024-09-16T08:26:02.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:26:02.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-124554051606 2024-09-16T08:26:02.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:26:02.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:26:02.782 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-124554051606 2024-09-16T08:26:02.782 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:26:02.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=124554051606 2024-09-16T08:26:02.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 124554051606' 2024-09-16T08:26:02.783 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 124554051606 2024-09-16T08:26:02.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:26:03.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 124554051607 -lt 124554051606 2024-09-16T08:26:03.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:26:03.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:26:03.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:26:03.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:26:03.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:26:03.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:26:03.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:26:03.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:26:03.788 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:26:03.788 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:26:03.788 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:26:04.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:26:04.215 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:26:04.216 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:26:04.216 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:26:04.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:26:04.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:26:04.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:26:04.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-remove-79228-1 2024-09-16T08:26:04.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T08:26:04.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T08:26:04.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-remove-79228-1 2024-09-16T08:26:04.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-16T08:26:04.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-16T08:26:04.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-remove-79228-1 td/test-erasure-eio/COPY 2024-09-16T08:26:04.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-16T08:26:04.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-16T08:26:04.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2024-09-16T08:26:04.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 1 + 1 2024-09-16T08:26:04.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: shard_id=2 2024-09-16T08:26:04.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_remove ec data pool-jerasure obj-remove-79228-1 td/test-erasure-eio 2 2024-09-16T08:26:04.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:127: inject_remove: local pooltype=ec 2024-09-16T08:26:04.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:128: inject_remove: shift 2024-09-16T08:26:04.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:129: inject_remove: local which=data 2024-09-16T08:26:04.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:130: inject_remove: shift 2024-09-16T08:26:04.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:131: inject_remove: local poolname=pool-jerasure 2024-09-16T08:26:04.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:132: inject_remove: shift 2024-09-16T08:26:04.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:133: inject_remove: local objname=obj-remove-79228-1 2024-09-16T08:26:04.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:134: inject_remove: shift 2024-09-16T08:26:04.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:135: inject_remove: local dir=td/test-erasure-eio 2024-09-16T08:26:04.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:136: inject_remove: shift 2024-09-16T08:26:04.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:137: inject_remove: local shard_id=2 2024-09-16T08:26:04.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:138: inject_remove: shift 2024-09-16T08:26:04.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:26:04.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: get_osds pool-jerasure obj-remove-79228-1 2024-09-16T08:26:04.851 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:26:04.851 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-remove-79228-1 2024-09-16T08:26:04.851 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-remove-79228-1 2024-09-16T08:26:04.851 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:26:05.254 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:26:05.254 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:26:05.254 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T08:26:05.254 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-16T08:26:05.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: local -a initial_osds 2024-09-16T08:26:05.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:141: inject_remove: local osd_id=2 2024-09-16T08:26:05.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:142: inject_remove: objectstore_tool td/test-erasure-eio 2 obj-remove-79228-1 remove 2024-09-16T08:26:05.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-16T08:26:05.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T08:26:05.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-16T08:26:05.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T08:26:05.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-remove-79228-1 remove 2024-09-16T08:26:05.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-16T08:26:05.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T08:26:05.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-16T08:26:05.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T08:26:05.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-16T08:26:05.257 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:26:05.257 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:26:05.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:26:05.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:26:05.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:26:06.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:26:06.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-remove-79228-1 remove 2024-09-16T08:26:06.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-16T08:26:06.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T08:26:06.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-16T08:26:06.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T08:26:06.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-16T08:26:06.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-remove-79228-1 remove 2024-09-16T08:26:07.981 INFO:tasks.workunit.client.0.smithi162.stdout:remove 2#2:bc16985e:::obj-remove-79228-1:head# 2024-09-16T08:26:08.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-16T08:26:08.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-16T08:26:08.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T08:26:08.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T08:26:08.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T08:26:08.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-16T08:26:08.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:26:08.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:26:08.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:26:08.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:26:08.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-16T08:26:08.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-16T08:26:08.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T08:26:08.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T08:26:08.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:26:08.553 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T08:26:08.553 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:26:08.553 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:26:08.554 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:26:08.554 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:26:08.554 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:26:08.554 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:26:08.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:26:08.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:26:08.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:26:08.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:26:08.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:26:08.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:26:08.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:26:08.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:26:08.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T08:26:08.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T08:26:08.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-16T08:26:08.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T08:26:08.557 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.2 2024-09-16T08:26:08.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:26:08.558 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-16T08:26:08.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T08:26:08.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T08:26:08.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T08:26:08.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T08:26:08.595 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:26:08.593+0000 7fd956274880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:26:08.599 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:26:08.597+0000 7fd956274880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:26:08.603 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:26:08.601+0000 7fd956274880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:26:08.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T08:26:08.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:26:08.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T08:26:08.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:26:08.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:26:08.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:26:08.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:26:08.996 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-16T08:26:08.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:26:08.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:26:09.168 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:26:09.165+0000 7fd956274880 -1 Falling back to public interface 2024-09-16T08:26:09.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:26:10.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:26:10.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:26:10.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:26:10.446 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:26:10.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:26:10.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:26:10.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:26:11.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:26:11.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:26:11.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:26:11.872 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:26:11.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:26:11.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:26:11.993 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:26:11.989+0000 7fd956274880 -1 osd.2 45 log_to_monitors true 2024-09-16T08:26:12.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:26:13.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:26:13.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:26:13.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:26:13.335 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-16T08:26:13.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:26:13.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:26:13.777 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 up in weight 1 up_from 50 up_thru 39 down_at 46 last_clean_interval [19,45) [v2:127.0.0.1:6818/3375156177,v1:127.0.0.1:6819/3375156177] [v2:127.0.0.1:6820/3375156177,v1:127.0.0.1:6821/3375156177] exists,up 1c5ce5de-ba81-4b6a-92a7-396142aeb579 2024-09-16T08:26:13.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:26:13.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:26:13.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:26:13.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-16T08:26:13.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:26:13.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:26:13.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:26:13.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:26:13.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:26:13.779 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:26:13.779 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:26:13.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:26:13.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:26:13.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:26:13.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:26:13.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:26:13.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:26:13.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:26:13.989 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:26:14.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:26:14.492 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:26:14.492 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:26:14.492 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:26:14.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:26:14.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:26:14.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:26:14.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803823 2024-09-16T08:26:14.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803823 2024-09-16T08:26:14.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803823' 2024-09-16T08:26:14.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:26:14.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:26:14.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=184683593736 2024-09-16T08:26:14.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 184683593736 2024-09-16T08:26:14.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803823 1-184683593736' 2024-09-16T08:26:14.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:26:14.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:26:15.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=214748364803 2024-09-16T08:26:15.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 214748364803 2024-09-16T08:26:15.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803823 1-184683593736 2-214748364803' 2024-09-16T08:26:15.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:26:15.011 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:26:15.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=124554051611 2024-09-16T08:26:15.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 124554051611 2024-09-16T08:26:15.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803823 1-184683593736 2-214748364803 3-124554051611' 2024-09-16T08:26:15.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:26:15.168 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803823 2024-09-16T08:26:15.168 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:26:15.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:26:15.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803823 2024-09-16T08:26:15.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:26:15.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803823 2024-09-16T08:26:15.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803823' 2024-09-16T08:26:15.173 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803823 2024-09-16T08:26:15.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:26:15.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803823 -lt 25769803823 2024-09-16T08:26:15.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:26:15.645 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-184683593736 2024-09-16T08:26:15.645 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:26:15.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:26:15.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-184683593736 2024-09-16T08:26:15.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:26:15.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=184683593736 2024-09-16T08:26:15.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 184683593736' 2024-09-16T08:26:15.651 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 184683593736 2024-09-16T08:26:15.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:26:16.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 184683593736 -lt 184683593736 2024-09-16T08:26:16.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:26:16.168 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-214748364803 2024-09-16T08:26:16.169 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:26:16.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:26:16.171 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-214748364803 2024-09-16T08:26:16.171 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:26:16.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=214748364803 2024-09-16T08:26:16.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 214748364803' 2024-09-16T08:26:16.174 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 214748364803 2024-09-16T08:26:16.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:26:16.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 214748364803 -lt 214748364803 2024-09-16T08:26:16.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:26:16.721 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-124554051611 2024-09-16T08:26:16.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:26:16.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:26:16.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-124554051611 2024-09-16T08:26:16.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:26:16.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=124554051611 2024-09-16T08:26:16.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 124554051611' 2024-09-16T08:26:16.727 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 124554051611 2024-09-16T08:26:16.728 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:26:17.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 124554051611 -lt 124554051611 2024-09-16T08:26:17.226 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:26:17.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:26:17.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:26:17.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:26:17.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:26:17.763 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:26:17.763 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:26:17.763 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:26:17.763 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:26:17.763 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:26:17.763 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:26:18.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:26:18.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:26:18.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:26:18.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:26:18.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:26:18.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:26:18.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:26:18.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-remove-79228-1 fail 2024-09-16T08:26:18.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T08:26:18.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T08:26:18.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-remove-79228-1 2024-09-16T08:26:18.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-16T08:26:18.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-16T08:26:18.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-remove-79228-1 td/test-erasure-eio/COPY 2024-09-16T08:26:18.769 INFO:tasks.workunit.client.0.smithi162.stderr:error getting pool-jerasure/obj-remove-79228-1: (5) Input/output error 2024-09-16T08:26:18.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-16T08:26:18.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:184: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2024-09-16T08:26:18.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:290: TEST_rados_get_subread_missing: delete_erasure_coded_pool pool-jerasure 2024-09-16T08:26:18.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T08:26:18.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-16T08:26:19.518 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-16T08:26:19.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-16T08:26:20.400 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-16T08:26:20.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-16T08:26:20.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T08:26:20.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T08:26:20.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T08:26:20.415 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:26:20.415 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:26:20.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:26:20.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:26:20.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:26:20.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:26:20.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T08:26:20.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T08:26:20.536 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T08:26:20.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T08:26:20.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T08:26:20.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T08:26:20.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:26:20.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T08:26:20.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T08:26:20.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:26:20.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:26:20.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T08:26:20.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T08:26:20.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T08:26:20.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:26:20.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:26:20.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T08:26:20.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T08:26:20.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T08:26:20.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-16T08:26:20.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-16T08:26:20.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-16T08:26:20.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-16T08:26:20.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T08:26:20.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T08:26:20.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T08:26:20.569 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:26:20.569 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:26:20.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:26:20.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:26:20.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:26:20.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:26:20.574 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T08:26:20.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T08:26:20.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T08:26:20.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T08:26:20.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T08:26:20.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T08:26:20.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:26:20.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T08:26:20.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T08:26:20.581 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:26:20.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:26:20.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T08:26:20.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T08:26:20.587 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T08:26:20.587 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:26:20.587 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:26:20.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T08:26:20.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T08:26:20.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T08:26:20.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-16T08:26:20.591 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T08:26:20.591 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:26:20.591 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:26:20.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79228 2024-09-16T08:26:20.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T08:26:20.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T08:26:20.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T08:26:20.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-16T08:26:20.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-16T08:26:20.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-16T08:26:20.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T08:26:20.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T08:26:20.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T08:26:20.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-16T08:26:20.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-16T08:26:21.035 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T08:26:21.036 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:26:21.036 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:26:21.036 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:26:21.036 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:26:21.037 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:26:21.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:26:21.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T08:26:21.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T08:26:21.258 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T08:26:21.258 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T08:26:21.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T08:26:21.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T08:26:21.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T08:26:21.259 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T08:26:21.260 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T08:26:21.260 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T08:26:21.260 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:26:21.260 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:26:21.260 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:26:21.261 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T08:26:21.261 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T08:26:21.261 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get fsid 2024-09-16T08:26:21.358 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T08:26:21.358 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T08:26:21.358 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T08:26:21.359 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T08:26:21.359 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T08:26:21.360 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T08:26:21.360 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T08:26:21.360 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T08:26:21.360 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:26:21.360 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:26:21.360 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:26:21.360 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T08:26:21.360 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T08:26:21.361 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get mon_host 2024-09-16T08:26:21.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-16T08:26:21.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-16T08:26:21.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T08:26:21.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T08:26:21.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T08:26:21.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-16T08:26:21.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T08:26:22.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T08:26:22.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:26:22.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:26:22.366 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:26:22.366 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:26:22.366 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:26:22.366 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:26:22.367 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T08:26:22.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T08:26:22.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-16T08:26:22.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T08:26:22.749 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-16T08:26:22.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T08:26:23.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-16T08:26:23.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T08:26:23.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T08:26:23.762 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:26:23.762 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:26:23.762 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:26:23.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T08:26:23.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-16T08:26:23.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-mon.a.asok log flush 2024-09-16T08:26:23.849 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T08:26:23.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-16T08:26:23.861 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T08:26:21.089+0000 7f53a2b70d40 0 load: jerasure load: lrc 2024-09-16T08:26:23.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_with_subreadall_eio_shard_0 td/test-erasure-eio 2024-09-16T08:26:23.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:330: TEST_rados_get_with_subreadall_eio_shard_0: local dir=td/test-erasure-eio 2024-09-16T08:26:23.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:331: TEST_rados_get_with_subreadall_eio_shard_0: local shard_id=0 2024-09-16T08:26:23.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:333: TEST_rados_get_with_subreadall_eio_shard_0: setup_osds 4 2024-09-16T08:26:23.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-16T08:26:23.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-16T08:26:23.863 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-16T08:26:23.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-16T08:26:23.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:26:23.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-16T08:26:23.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:26:23.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:26:23.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T08:26:23.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:26:23.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-16T08:26:23.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:26:23.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:26:23.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:26:23.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:26:23.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-16T08:26:23.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-16T08:26:23.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:26:23.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:26:23.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:26:23.868 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:26:23.868 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:26:23.868 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:26:23.868 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:26:23.868 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:26:23.868 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:26:23.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:26:23.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:26:23.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:26:23.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:26:23.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:26:23.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:26:23.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:26:23.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:26:23.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:26:23.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:26:23.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:26:23.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:26:23.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:26:23.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-16T08:26:23.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:26:23.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0f11253b-564a-4c2e-b3a2-26d811b19437 2024-09-16T08:26:23.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0f11253b-564a-4c2e-b3a2-26d811b19437' 2024-09-16T08:26:23.872 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 0f11253b-564a-4c2e-b3a2-26d811b19437 2024-09-16T08:26:23.872 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:26:23.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCv6+dm3C6VNRAAPIyxkG5O4W4mFc6hI66oRQ== 2024-09-16T08:26:23.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCv6+dm3C6VNRAAPIyxkG5O4W4mFc6hI66oRQ=="}' 2024-09-16T08:26:23.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0f11253b-564a-4c2e-b3a2-26d811b19437 -i td/test-erasure-eio/0/new.json 2024-09-16T08:26:24.254 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:26:24.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-16T08:26:24.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCv6+dm3C6VNRAAPIyxkG5O4W4mFc6hI66oRQ== --osd-uuid 0f11253b-564a-4c2e-b3a2-26d811b19437 2024-09-16T08:26:24.306 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:26:24.304+0000 7fa560e7c880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:26:24.310 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:26:24.304+0000 7fa560e7c880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:26:24.312 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:26:24.308+0000 7fa560e7c880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:26:24.313 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:26:24.308+0000 7fa560e7c880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T08:26:24.313 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:26:24.308+0000 7fa560e7c880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T08:26:24.313 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:26:24.308+0000 7fa560e7c880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T08:26:24.411 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:26:24.408+0000 7fa560e7c880 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-16T08:26:30.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-16T08:26:30.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:26:30.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T08:26:30.113 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T08:26:30.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:26:30.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T08:26:30.669 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T08:26:30.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:26:30.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:26:30.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:26:30.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:26:30.702 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:26:30.700+0000 7f9c515bc880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:26:30.705 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:26:30.700+0000 7f9c515bc880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:26:30.706 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:26:30.704+0000 7f9c515bc880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:26:31.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T08:26:31.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:26:31.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T08:26:31.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:26:31.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:26:31.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:26:31.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:26:31.166 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:26:31.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:26:31.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:26:31.256 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:26:31.252+0000 7f9c515bc880 -1 Falling back to public interface 2024-09-16T08:26:31.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:26:32.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:26:32.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:26:32.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:26:32.607 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:26:32.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:26:32.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:26:33.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:26:33.657 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:26:33.652+0000 7f9c515bc880 -1 osd.0 0 log_to_monitors true 2024-09-16T08:26:34.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:26:34.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:26:34.037 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:26:34.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:26:34.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:26:34.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:26:34.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:26:35.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:26:35.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:26:35.639 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:26:35.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:26:35.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:26:35.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:26:36.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:26:37.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:26:37.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:26:37.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:26:37.075 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:26:37.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:26:37.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:26:37.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:26:38.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:26:38.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:26:38.493 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:26:38.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:26:38.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:26:38.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:26:38.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:26:39.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:26:39.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:26:39.916 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:26:39.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:26:39.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:26:39.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:26:40.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:26:41.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:26:41.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:26:41.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:26:41.454 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:26:41.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:26:41.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:26:41.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:26:42.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:26:42.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:26:42.887 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:26:42.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:26:42.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:26:42.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:26:43.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:26:44.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:26:44.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:26:44.378 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:26:44.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:26:44.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:26:44.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:26:44.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:26:45.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:26:45.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:26:45.812 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:26:45.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:26:45.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:26:45.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:26:46.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:26:47.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:26:47.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:26:47.357 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:26:47.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:26:47.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:26:47.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:26:48.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:26:49.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:26:49.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:26:49.251 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:26:49.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:26:49.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:26:49.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:26:50.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:26:51.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:26:51.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:26:51.077 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:26:51.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:26:51.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:26:51.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:26:52.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:26:53.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:26:53.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:26:53.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:26:53.078 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:26:53.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:26:53.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:26:54.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:26:55.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:26:55.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:26:55.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:26:55.203 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:26:55.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:26:55.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:26:56.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:26:57.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:26:57.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:26:57.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:26:57.307 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:26:57.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:26:57.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:26:57.895 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2390584637,v1:127.0.0.1:6803/2390584637] [v2:127.0.0.1:6804/2390584637,v1:127.0.0.1:6805/2390584637] exists,up 0f11253b-564a-4c2e-b3a2-26d811b19437 2024-09-16T08:26:57.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:26:57.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:26:57.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:26:57.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:26:57.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-16T08:26:57.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:26:57.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:26:57.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T08:26:57.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:26:57.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-16T08:26:57.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:26:57.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:26:57.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:26:57.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:26:57.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-16T08:26:57.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-16T08:26:57.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:26:57.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:26:57.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:26:57.900 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:26:57.900 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:26:57.900 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:26:57.900 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:26:57.900 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:26:57.900 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:26:57.902 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:26:57.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:26:57.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:26:57.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:26:57.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:26:57.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:26:57.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:26:57.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:26:57.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:26:57.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:26:57.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:26:57.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:26:57.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:26:57.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-16T08:26:57.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:26:57.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5e11ab99-d91c-4c55-b06b-13f3e9efaef3 2024-09-16T08:26:57.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5e11ab99-d91c-4c55-b06b-13f3e9efaef3' 2024-09-16T08:26:57.904 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 5e11ab99-d91c-4c55-b06b-13f3e9efaef3 2024-09-16T08:26:57.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:26:57.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDR6+dmoW1zNxAANNWRLq+a2i4kN3tQeDFf9A== 2024-09-16T08:26:57.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDR6+dmoW1zNxAANNWRLq+a2i4kN3tQeDFf9A=="}' 2024-09-16T08:26:57.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5e11ab99-d91c-4c55-b06b-13f3e9efaef3 -i td/test-erasure-eio/1/new.json 2024-09-16T08:26:58.583 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:26:58.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-16T08:26:58.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDR6+dmoW1zNxAANNWRLq+a2i4kN3tQeDFf9A== --osd-uuid 5e11ab99-d91c-4c55-b06b-13f3e9efaef3 2024-09-16T08:26:58.636 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:26:58.632+0000 7f2146e05880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:26:58.665 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:26:58.660+0000 7f2146e05880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:26:58.668 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:26:58.664+0000 7f2146e05880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:26:58.668 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:26:58.664+0000 7f2146e05880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T08:26:58.668 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:26:58.664+0000 7f2146e05880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T08:26:58.669 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:26:58.664+0000 7f2146e05880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T08:26:58.756 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:26:58.752+0000 7f2146e05880 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-16T08:27:04.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-16T08:27:04.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:27:04.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T08:27:04.677 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-16T08:27:04.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:27:05.229 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-16T08:27:05.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T08:27:05.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:27:05.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:27:05.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:27:05.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:27:05.265 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:27:05.260+0000 7f547f9c0880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:27:05.288 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:27:05.284+0000 7f547f9c0880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:27:05.289 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:27:05.284+0000 7f547f9c0880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:27:05.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T08:27:05.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:27:05.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T08:27:05.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:27:05.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:27:05.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:05.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:27:05.661 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:27:05.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:05.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:27:05.840 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:27:05.836+0000 7f547f9c0880 -1 Falling back to public interface 2024-09-16T08:27:06.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:07.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:07.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:07.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:27:07.102 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:27:07.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:07.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:27:07.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:08.440 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:27:08.436+0000 7f547f9c0880 -1 osd.1 0 log_to_monitors true 2024-09-16T08:27:08.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:08.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:08.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:27:08.526 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:27:08.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:08.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:27:09.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:10.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:10.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:10.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:27:10.130 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:27:10.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:10.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:27:10.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:11.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:11.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:11.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:27:11.750 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:27:11.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:11.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:27:12.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:13.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:13.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:13.297 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:27:13.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:27:13.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:13.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:27:13.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:14.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:14.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:14.762 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:27:14.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:27:14.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:14.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:27:15.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:16.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:16.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:16.283 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:27:16.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:27:16.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:16.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:27:16.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:17.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:17.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:17.765 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:27:17.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:27:17.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:17.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:27:18.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:19.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:19.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:19.332 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:27:19.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:27:19.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:19.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:27:19.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:20.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:20.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:20.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:27:20.797 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:27:20.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:20.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:27:21.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:22.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:22.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:22.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:27:22.202 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:27:22.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:22.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:27:23.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:24.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:24.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:24.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:27:24.168 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:27:24.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:24.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:27:24.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:25.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:25.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:25.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:27:25.944 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:27:25.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:25.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:27:26.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:27.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:27.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:27.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:27:27.903 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:27:27.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:27.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:27:28.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:29.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:29.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:29.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:27:29.747 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:27:29.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:29.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:27:30.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:31.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:31.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:31.820 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:27:31.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:27:31.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:31.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:27:32.367 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 14 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/162298052,v1:127.0.0.1:6811/162298052] [v2:127.0.0.1:6812/162298052,v1:127.0.0.1:6813/162298052] exists,up 5e11ab99-d91c-4c55-b06b-13f3e9efaef3 2024-09-16T08:27:32.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:27:32.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:27:32.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:27:32.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:27:32.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-16T08:27:32.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:27:32.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:27:32.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T08:27:32.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:27:32.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-16T08:27:32.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:27:32.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:27:32.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:27:32.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:27:32.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-16T08:27:32.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-16T08:27:32.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:27:32.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:27:32.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:27:32.372 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:27:32.372 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:27:32.372 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:27:32.372 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:27:32.372 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:27:32.372 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:27:32.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:27:32.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:27:32.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:27:32.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:27:32.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:27:32.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:27:32.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:27:32.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:27:32.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:27:32.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:27:32.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:27:32.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:27:32.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:27:32.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-16T08:27:32.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:27:32.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bf6783b4-4e22-4a69-99ff-beaeba361375 2024-09-16T08:27:32.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 bf6783b4-4e22-4a69-99ff-beaeba361375' 2024-09-16T08:27:32.376 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 bf6783b4-4e22-4a69-99ff-beaeba361375 2024-09-16T08:27:32.376 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:27:32.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD06+dmW6v7FxAAo9mQfsugF1A0fihagAyc0Q== 2024-09-16T08:27:32.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD06+dmW6v7FxAAo9mQfsugF1A0fihagAyc0Q=="}' 2024-09-16T08:27:32.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bf6783b4-4e22-4a69-99ff-beaeba361375 -i td/test-erasure-eio/2/new.json 2024-09-16T08:27:33.139 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:27:33.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-16T08:27:33.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD06+dmW6v7FxAAo9mQfsugF1A0fihagAyc0Q== --osd-uuid bf6783b4-4e22-4a69-99ff-beaeba361375 2024-09-16T08:27:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:27:33.187+0000 7f0f90718880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:27:33.220 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:27:33.215+0000 7f0f90718880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:27:33.222 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:27:33.219+0000 7f0f90718880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:27:33.223 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:27:33.219+0000 7f0f90718880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:27:33.223 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:27:33.219+0000 7f0f90718880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:27:33.223 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:27:33.219+0000 7f0f90718880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:27:33.311 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:27:33.307+0000 7f0f90718880 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-16T08:27:39.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-16T08:27:39.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:27:39.485 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-16T08:27:39.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T08:27:39.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:27:40.095 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-16T08:27:40.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T08:27:40.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:27:40.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:27:40.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:27:40.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:27:40.131 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:27:40.127+0000 7f31ab76a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:27:40.159 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:27:40.155+0000 7f31ab76a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:27:40.161 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:27:40.155+0000 7f31ab76a880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:27:40.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T08:27:40.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:27:40.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T08:27:40.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:27:40.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:27:40.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:40.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:27:40.582 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:27:40.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:40.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:27:40.724 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:27:40.719+0000 7f31ab76a880 -1 Falling back to public interface 2024-09-16T08:27:41.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:42.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:42.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:42.018 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:27:42.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:27:42.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:42.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:27:42.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:43.295 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:27:43.291+0000 7f31ab76a880 -1 osd.2 0 log_to_monitors true 2024-09-16T08:27:43.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:43.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:43.445 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:27:43.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:27:43.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:43.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:27:43.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:44.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:44.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:44.986 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:27:44.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:27:44.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:44.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:27:45.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:46.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:46.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:46.429 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:27:46.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:27:46.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:46.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:27:46.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:47.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:47.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:47.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:27:47.906 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:27:47.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:47.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:27:48.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:49.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:49.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:49.481 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:27:49.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:27:49.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:49.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:27:49.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:50.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:50.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:50.911 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:27:50.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:27:50.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:50.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:27:51.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:52.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:52.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:52.430 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:27:52.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:27:52.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:52.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:27:52.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:53.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:53.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:53.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:27:53.912 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:27:53.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:53.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:27:54.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:55.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:55.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:55.484 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:27:55.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:27:55.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:55.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:27:55.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:56.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:56.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:56.892 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:27:56.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:27:56.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:56.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:27:57.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:27:58.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:27:58.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:27:58.611 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:27:58.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:27:58.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:27:58.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:27:59.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:28:00.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:28:00.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:28:00.136 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:28:00.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:28:00.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:28:00.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:28:00.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:28:01.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:28:01.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:28:01.894 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:28:01.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:28:01.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:28:01.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:28:02.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:28:03.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:28:03.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:28:03.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:28:03.894 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:28:03.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:28:03.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:28:04.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:28:05.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:28:05.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:28:05.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:28:05.754 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:28:05.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:28:05.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:28:06.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:28:07.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:28:07.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:28:07.595 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:28:07.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:28:07.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:28:07.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:28:08.022 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/646620552,v1:127.0.0.1:6819/646620552] [v2:127.0.0.1:6820/646620552,v1:127.0.0.1:6821/646620552] exists,up bf6783b4-4e22-4a69-99ff-beaeba361375 2024-09-16T08:28:08.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:28:08.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:28:08.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:28:08.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:28:08.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-16T08:28:08.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:28:08.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:28:08.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T08:28:08.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:28:08.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-16T08:28:08.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:28:08.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:28:08.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:28:08.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:28:08.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-16T08:28:08.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-16T08:28:08.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:28:08.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:28:08.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:28:08.026 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:28:08.026 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:28:08.026 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:28:08.026 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:28:08.026 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:28:08.027 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:28:08.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:28:08.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:28:08.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:28:08.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:28:08.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:28:08.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:28:08.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:28:08.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:28:08.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:28:08.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:28:08.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:28:08.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:28:08.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:28:08.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-16T08:28:08.030 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:28:08.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e3c5da55-d96f-419d-9699-1f06f4ce630f 2024-09-16T08:28:08.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 e3c5da55-d96f-419d-9699-1f06f4ce630f' 2024-09-16T08:28:08.031 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 e3c5da55-d96f-419d-9699-1f06f4ce630f 2024-09-16T08:28:08.031 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:28:08.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAY7Odm6dsgAxAAUE9NPR1Z4HtBaSfiSX1cbA== 2024-09-16T08:28:08.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAY7Odm6dsgAxAAUE9NPR1Z4HtBaSfiSX1cbA=="}' 2024-09-16T08:28:08.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e3c5da55-d96f-419d-9699-1f06f4ce630f -i td/test-erasure-eio/3/new.json 2024-09-16T08:28:08.978 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:28:08.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-16T08:28:08.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAY7Odm6dsgAxAAUE9NPR1Z4HtBaSfiSX1cbA== --osd-uuid e3c5da55-d96f-419d-9699-1f06f4ce630f 2024-09-16T08:28:09.031 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:28:09.026+0000 7f7723b22880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:28:09.043 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:28:09.038+0000 7f7723b22880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:28:09.045 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:28:09.038+0000 7f7723b22880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:28:09.046 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:28:09.042+0000 7f7723b22880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:28:09.046 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:28:09.042+0000 7f7723b22880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:28:09.046 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:28:09.042+0000 7f7723b22880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:28:09.167 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:28:09.166+0000 7f7723b22880 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-16T08:28:15.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-16T08:28:15.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:28:15.633 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-16T08:28:15.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T08:28:15.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:28:16.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T08:28:16.187 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-16T08:28:16.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:28:16.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:28:16.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:28:16.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:28:16.221 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:28:16.218+0000 7feca0b52880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:28:16.223 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:28:16.222+0000 7feca0b52880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:28:16.226 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:28:16.226+0000 7feca0b52880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:28:16.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T08:28:16.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:28:16.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T08:28:16.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:28:16.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:28:16.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:28:16.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:28:16.654 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:28:16.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:28:16.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:28:16.800 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:28:16.798+0000 7feca0b52880 -1 Falling back to public interface 2024-09-16T08:28:17.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:28:18.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:28:18.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:28:18.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:28:18.095 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:28:18.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:28:18.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:28:18.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:28:19.123 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:28:19.122+0000 7feca0b52880 -1 osd.3 0 log_to_monitors true 2024-09-16T08:28:19.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:28:19.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:28:19.549 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:28:19.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:28:19.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:28:19.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:28:20.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:28:21.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:28:21.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:28:21.527 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:28:21.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:28:21.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:28:21.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:28:22.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:28:23.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:28:23.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:28:23.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:28:23.192 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:28:23.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:28:23.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:28:23.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:28:24.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:28:24.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:28:24.655 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:28:24.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:28:24.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:28:24.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:28:25.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:28:26.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:28:26.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:28:26.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:28:26.206 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:28:26.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:28:26.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:28:26.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:28:27.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:28:27.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:28:27.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:28:27.781 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:28:27.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:28:27.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:28:28.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:28:29.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:28:29.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:28:29.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:28:29.276 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:28:29.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:28:29.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:28:29.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:28:30.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:28:30.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:28:30.728 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:28:30.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:28:30.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:28:30.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:28:31.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:28:32.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:28:32.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:28:32.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:28:32.197 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:28:32.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:28:32.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:28:32.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:28:33.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:28:33.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:28:33.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:28:33.738 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:28:33.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:28:33.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:28:34.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:28:35.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:28:35.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:28:35.716 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:28:35.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:28:35.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:28:35.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:28:36.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:28:37.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:28:37.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:28:37.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:28:37.498 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:28:37.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:28:37.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:28:38.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:28:39.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:28:39.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:28:39.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:28:39.199 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:28:39.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:28:39.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:28:39.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:28:40.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:28:40.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:28:40.912 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:28:40.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:28:40.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:28:40.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:28:41.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:28:42.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:28:42.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:28:42.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:28:42.844 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:28:42.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:28:42.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:28:43.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:28:44.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:28:44.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:28:44.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:28:44.280 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:28:44.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:28:44.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:28:44.957 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3239501563,v1:127.0.0.1:6827/3239501563] [v2:127.0.0.1:6828/3239501563,v1:127.0.0.1:6829/3239501563] exists,up e3c5da55-d96f-419d-9699-1f06f4ce630f 2024-09-16T08:28:44.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:28:44.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:28:44.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:28:44.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-16T08:28:44.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T08:28:44.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T08:28:44.960 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:28:44.960 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:28:44.960 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:28:44.960 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.0.asok 2024-09-16T08:28:44.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-16T08:28:44.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.0.asok log flush 2024-09-16T08:28:45.055 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T08:28:45.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-16T08:28:45.070 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T08:26:31.268+0000 7f9c515bc880 0 load: jerasure load: lrc 2024-09-16T08:28:45.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:335: TEST_rados_get_with_subreadall_eio_shard_0: local poolname=pool-jerasure 2024-09-16T08:28:45.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:336: TEST_rados_get_with_subreadall_eio_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2024-09-16T08:28:45.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T08:28:45.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-16T08:28:45.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-16T08:28:45.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-16T08:28:45.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-16T08:28:45.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-16T08:28:45.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-16T08:28:46.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-16T08:28:46.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-16T08:28:46.940 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-16T08:28:46.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T08:28:47.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-16T08:28:47.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:28:47.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:28:47.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:28:47.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:28:47.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:28:47.953 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:28:47.953 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:28:47.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:28:47.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:28:47.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:28:48.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:28:48.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:28:48.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:28:48.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:28:48.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:28:48.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:28:48.622 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:28:48.622 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:28:48.622 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:28:48.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:28:48.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:28:48.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:28:48.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803809 2024-09-16T08:28:48.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803809 2024-09-16T08:28:48.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803809' 2024-09-16T08:28:48.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:28:48.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:28:49.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=60129542169 2024-09-16T08:28:49.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 60129542169 2024-09-16T08:28:49.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803809 1-60129542169' 2024-09-16T08:28:49.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:28:49.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:28:49.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345949 2024-09-16T08:28:49.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345949 2024-09-16T08:28:49.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803809 1-60129542169 2-85899345949' 2024-09-16T08:28:49.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:28:49.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:28:49.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=128849018895 2024-09-16T08:28:49.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 128849018895 2024-09-16T08:28:49.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803809 1-60129542169 2-85899345949 3-128849018895' 2024-09-16T08:28:49.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:28:49.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803809 2024-09-16T08:28:49.370 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:28:49.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:28:49.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803809 2024-09-16T08:28:49.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:28:49.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803809 2024-09-16T08:28:49.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803809' 2024-09-16T08:28:49.376 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803809 2024-09-16T08:28:49.376 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:28:49.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803806 -lt 25769803809 2024-09-16T08:28:49.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T08:28:50.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T08:28:50.843 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:28:51.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803809 -lt 25769803809 2024-09-16T08:28:51.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:28:51.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-60129542169 2024-09-16T08:28:51.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:28:51.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:28:51.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-60129542169 2024-09-16T08:28:51.332 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:28:51.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=60129542169 2024-09-16T08:28:51.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 60129542169' 2024-09-16T08:28:51.335 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 60129542169 2024-09-16T08:28:51.335 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:28:51.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 60129542169 -lt 60129542169 2024-09-16T08:28:51.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:28:51.825 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-85899345949 2024-09-16T08:28:51.825 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:28:51.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:28:51.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-85899345949 2024-09-16T08:28:51.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:28:51.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345949 2024-09-16T08:28:51.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 85899345949' 2024-09-16T08:28:51.830 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 85899345949 2024-09-16T08:28:51.831 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:28:52.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345949 -lt 85899345949 2024-09-16T08:28:52.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:28:52.342 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-128849018895 2024-09-16T08:28:52.342 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:28:52.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:28:52.345 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-128849018895 2024-09-16T08:28:52.345 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:28:52.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=128849018895 2024-09-16T08:28:52.348 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 128849018895 2024-09-16T08:28:52.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 128849018895' 2024-09-16T08:28:52.348 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:28:52.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 128849018895 -lt 128849018895 2024-09-16T08:28:52.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:28:52.829 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:28:52.829 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:28:53.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:28:53.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:28:53.427 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:28:53.427 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:28:53.427 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:28:53.427 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:28:53.427 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:28:53.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:28:53.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:28:53.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:28:53.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:28:53.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:28:54.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:28:54.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:28:54.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:28:54.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:338: TEST_rados_get_with_subreadall_eio_shard_0: rados_put_get_data eio td/test-erasure-eio 0 recovery 2024-09-16T08:28:54.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-09-16T08:28:54.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-16T08:28:54.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-16T08:28:54.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-16T08:28:54.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=0 2024-09-16T08:28:54.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-16T08:28:54.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg=recovery 2024-09-16T08:28:54.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-16T08:28:54.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-79228-0 2024-09-16T08:28:54.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-79228-0 2024-09-16T08:28:54.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-16T08:28:54.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-16T08:28:54.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-79228-0 2024-09-16T08:28:54.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:28:54.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-16T08:28:54.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:28:54.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-16T08:28:54.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:28:54.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-16T08:28:54.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:28:54.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-16T08:28:54.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-79228-0 td/test-erasure-eio/ORIGINAL 2024-09-16T08:28:54.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-79228-0 td/test-erasure-eio 0 2024-09-16T08:28:54.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=ec 2024-09-16T08:28:54.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T08:28:54.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T08:28:54.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T08:28:54.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-jerasure 2024-09-16T08:28:54.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T08:28:54.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-eio-79228-0 2024-09-16T08:28:54.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T08:28:54.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/test-erasure-eio 2024-09-16T08:28:54.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T08:28:54.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2024-09-16T08:28:54.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T08:28:54.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:28:54.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-jerasure obj-eio-79228-0 2024-09-16T08:28:54.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:28:54.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-79228-0 2024-09-16T08:28:54.595 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-79228-0 2024-09-16T08:28:54.595 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:28:55.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:28:55.070 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:28:55.070 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T08:28:55.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-16T08:28:55.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T08:28:55.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=3 2024-09-16T08:28:55.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' ec '!=' ec ']' 2024-09-16T08:28:55.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/test-erasure-eio/3/type 2024-09-16T08:28:55.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T08:28:55.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-16T08:28:55.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T08:28:55.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-16T08:28:55.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T08:28:55.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T08:28:55.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T08:28:55.074 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-16T08:28:55.074 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T08:28:55.075 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T08:28:55.075 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:28:55.076 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:28:55.076 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:28:55.076 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.3.asok 2024-09-16T08:28:55.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79228/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-16T08:28:55.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T08:28:55.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T08:28:55.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T08:28:55.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.3 2024-09-16T08:28:55.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-16T08:28:55.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-16T08:28:55.173 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:28:55.173 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:28:55.173 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:28:55.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.3.asok 2024-09-16T08:28:55.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T08:28:55.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.3.asok injectdataerr pool-jerasure obj-eio-79228-0 0 2024-09-16T08:28:55.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-79228-0 2024-09-16T08:28:55.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T08:28:55.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T08:28:55.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-79228-0 2024-09-16T08:28:55.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-16T08:28:55.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-16T08:28:55.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-79228-0 td/test-erasure-eio/COPY 2024-09-16T08:28:55.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-16T08:28:55.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-16T08:28:55.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' recovery = recovery ']' 2024-09-16T08:28:55.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:28:55.316 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: get_osds pool-jerasure obj-eio-79228-0 2024-09-16T08:28:55.316 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:28:55.316 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-79228-0 2024-09-16T08:28:55.317 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-79228-0 2024-09-16T08:28:55.317 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:28:55.737 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:28:55.738 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:28:55.738 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T08:28:55.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-16T08:28:55.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: local -a initial_osds 2024-09-16T08:28:55.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:171: rados_put_get_data: local last_osd=2 2024-09-16T08:28:55.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:173: rados_put_get_data: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-16T08:28:55.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:28:55.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:28:55.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:28:55.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:28:55.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:28:56.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:28:56.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:174: rados_put_get_data: ceph osd out 2 2024-09-16T08:28:56.899 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 is already out. 2024-09-16T08:28:56.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: get_osds pool-jerasure obj-eio-79228-0 2024-09-16T08:28:56.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: grep '\<2\>' 2024-09-16T08:28:56.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:28:56.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-79228-0 2024-09-16T08:28:56.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:28:56.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-79228-0 2024-09-16T08:28:57.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:28:57.348 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:28:57.348 INFO:tasks.workunit.client.0.smithi162.stderr:0' 2024-09-16T08:28:57.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T08:28:57.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:176: rados_put_get_data: ceph osd in 2 2024-09-16T08:28:58.274 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 is already in. 2024-09-16T08:28:58.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:177: rados_put_get_data: activate_osd td/test-erasure-eio 2 2024-09-16T08:28:58.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-16T08:28:58.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T08:28:58.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T08:28:58.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T08:28:58.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-16T08:28:58.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:28:58.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:28:58.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:28:58.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:28:58.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-16T08:28:58.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-16T08:28:58.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T08:28:58.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T08:28:58.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:28:58.289 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T08:28:58.289 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:28:58.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:28:58.290 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:28:58.290 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:28:58.290 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:28:58.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:28:58.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:28:58.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:28:58.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:28:58.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:28:58.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:28:58.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:28:58.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:28:58.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:28:58.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T08:28:58.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T08:28:58.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-16T08:28:58.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T08:28:58.292 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-16T08:28:58.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:28:58.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-16T08:28:58.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T08:28:58.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T08:28:58.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T08:28:58.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T08:28:58.326 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:28:58.325+0000 7f4e43313880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:28:58.333 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:28:58.329+0000 7f4e43313880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:28:58.335 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:28:58.333+0000 7f4e43313880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:28:58.880 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:28:58.877+0000 7f4e43313880 -1 Falling back to public interface 2024-09-16T08:28:58.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T08:28:58.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:28:58.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T08:28:58.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:28:58.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:28:58.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:28:58.936 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:28:58.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:28:58.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:28:58.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:28:59.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:29:00.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:29:00.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:29:00.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:29:00.337 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:29:00.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:29:00.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:29:00.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:29:01.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:29:01.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:29:01.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:29:01.774 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:29:01.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:29:01.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:29:01.828 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:01.825+0000 7f4e43313880 -1 osd.2 40 log_to_monitors true 2024-09-16T08:29:02.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:29:03.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:29:03.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:29:03.195 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:29:03.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:29:03.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:29:03.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:29:03.798 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 50 up_thru 21 down_at 41 last_clean_interval [20,40) [v2:127.0.0.1:6818/3627321632,v1:127.0.0.1:6819/3627321632] [v2:127.0.0.1:6820/3627321632,v1:127.0.0.1:6821/3627321632] exists,up bf6783b4-4e22-4a69-99ff-beaeba361375 2024-09-16T08:29:03.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:29:03.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:29:03.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:29:03.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:178: rados_put_get_data: wait_for_clean 2024-09-16T08:29:03.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:29:03.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:29:03.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:29:03.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:29:03.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:29:03.801 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:29:03.801 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:29:03.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:29:03.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:29:03.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:29:04.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:29:04.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:29:04.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:29:04.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:29:04.010 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:29:04.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:29:04.504 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:29:04.504 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:29:04.504 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:29:04.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:29:04.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:29:04.504 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:29:04.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803814 2024-09-16T08:29:04.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803814 2024-09-16T08:29:04.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803814' 2024-09-16T08:29:04.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:29:04.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:29:04.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=60129542174 2024-09-16T08:29:04.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 60129542174 2024-09-16T08:29:04.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803814 1-60129542174' 2024-09-16T08:29:04.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:29:04.815 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:29:04.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=214748364803 2024-09-16T08:29:04.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 214748364803 2024-09-16T08:29:04.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803814 1-60129542174 2-214748364803' 2024-09-16T08:29:04.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:29:04.965 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:29:05.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=128849018900 2024-09-16T08:29:05.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 128849018900 2024-09-16T08:29:05.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803814 1-60129542174 2-214748364803 3-128849018900' 2024-09-16T08:29:05.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:29:05.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803814 2024-09-16T08:29:05.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:29:05.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:29:05.122 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803814 2024-09-16T08:29:05.122 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:29:05.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803814 2024-09-16T08:29:05.123 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803814 2024-09-16T08:29:05.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803814' 2024-09-16T08:29:05.124 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:29:05.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803812 -lt 25769803814 2024-09-16T08:29:05.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T08:29:06.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T08:29:06.523 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:29:06.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803814 -lt 25769803814 2024-09-16T08:29:06.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:29:06.997 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-60129542174 2024-09-16T08:29:06.997 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:29:06.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:29:07.001 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-60129542174 2024-09-16T08:29:07.001 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:29:07.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=60129542174 2024-09-16T08:29:07.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 60129542174' 2024-09-16T08:29:07.003 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 60129542174 2024-09-16T08:29:07.003 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:29:07.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 60129542174 -lt 60129542174 2024-09-16T08:29:07.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:29:07.505 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-214748364803 2024-09-16T08:29:07.505 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:29:07.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:29:07.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-214748364803 2024-09-16T08:29:07.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:29:07.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=214748364803 2024-09-16T08:29:07.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 214748364803' 2024-09-16T08:29:07.511 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 214748364803 2024-09-16T08:29:07.511 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:29:08.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 214748364803 -lt 214748364803 2024-09-16T08:29:08.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:29:08.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-128849018900 2024-09-16T08:29:08.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:29:08.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:29:08.044 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-128849018900 2024-09-16T08:29:08.045 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:29:08.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=128849018900 2024-09-16T08:29:08.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 128849018900' 2024-09-16T08:29:08.047 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 128849018900 2024-09-16T08:29:08.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:29:08.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 128849018900 -lt 128849018900 2024-09-16T08:29:08.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:29:08.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:29:08.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:29:09.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:29:09.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:29:09.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:29:09.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:29:09.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:29:09.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:29:09.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:29:09.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:29:09.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:29:09.468 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:29:09.468 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:29:09.468 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:29:10.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:29:10.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:29:10.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:29:10.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:340: TEST_rados_get_with_subreadall_eio_shard_0: delete_erasure_coded_pool pool-jerasure 2024-09-16T08:29:10.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T08:29:10.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-16T08:29:10.660 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-16T08:29:10.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-16T08:29:11.388 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-16T08:29:11.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-16T08:29:11.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T08:29:11.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T08:29:11.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T08:29:11.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:29:11.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:29:11.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:29:11.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:29:11.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:29:11.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:29:11.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T08:29:11.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T08:29:11.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T08:29:11.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T08:29:11.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T08:29:11.728 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T08:29:11.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:29:11.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T08:29:11.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T08:29:11.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:29:11.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:29:11.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T08:29:11.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T08:29:11.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T08:29:11.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:29:11.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:29:11.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T08:29:11.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T08:29:11.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T08:29:11.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-16T08:29:11.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-16T08:29:11.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-16T08:29:11.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-16T08:29:11.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T08:29:11.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T08:29:11.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T08:29:11.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:29:11.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:29:11.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:29:11.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:29:11.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:29:11.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:29:11.771 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T08:29:11.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T08:29:11.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T08:29:11.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T08:29:11.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T08:29:11.775 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T08:29:11.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:29:11.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T08:29:11.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T08:29:11.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:29:11.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:29:11.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T08:29:11.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T08:29:11.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T08:29:11.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:29:11.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:29:11.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T08:29:11.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T08:29:11.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T08:29:11.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-16T08:29:11.788 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T08:29:11.788 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:29:11.788 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:29:11.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79228 2024-09-16T08:29:11.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T08:29:11.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T08:29:11.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T08:29:11.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-16T08:29:11.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-16T08:29:11.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-16T08:29:11.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T08:29:11.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T08:29:11.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T08:29:11.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-16T08:29:11.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-16T08:29:12.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T08:29:12.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:29:12.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:29:12.212 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:29:12.212 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:29:12.212 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:29:12.213 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:29:12.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T08:29:12.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T08:29:12.448 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T08:29:12.448 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T08:29:12.448 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T08:29:12.448 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T08:29:12.448 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T08:29:12.449 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T08:29:12.449 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T08:29:12.449 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T08:29:12.450 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:29:12.450 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:29:12.450 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:29:12.451 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T08:29:12.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T08:29:12.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get fsid 2024-09-16T08:29:12.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T08:29:12.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T08:29:12.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T08:29:12.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T08:29:12.553 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T08:29:12.553 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T08:29:12.553 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T08:29:12.553 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T08:29:12.554 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:29:12.554 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:29:12.554 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:29:12.554 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T08:29:12.555 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T08:29:12.555 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79228/ceph-mon.a.asok config get mon_host 2024-09-16T08:29:12.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-16T08:29:12.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-16T08:29:12.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T08:29:12.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T08:29:12.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T08:29:12.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-16T08:29:12.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T08:29:13.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T08:29:13.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:29:13.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:29:13.561 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:29:13.561 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:29:13.561 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:29:13.561 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:29:13.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T08:29:13.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T08:29:13.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-16T08:29:13.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T08:29:13.964 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-16T08:29:13.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T08:29:14.974 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-16T08:29:14.974 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T08:29:14.974 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T08:29:14.975 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:29:14.975 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:29:14.975 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:29:14.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-mon.a.asok 2024-09-16T08:29:14.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-16T08:29:14.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-mon.a.asok log flush 2024-09-16T08:29:15.059 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T08:29:15.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-16T08:29:15.070 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T08:29:12.265+0000 7f8d1b990d40 0 load: jerasure load: lrc 2024-09-16T08:29:15.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_with_subreadall_eio_shard_1 td/test-erasure-eio 2024-09-16T08:29:15.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:344: TEST_rados_get_with_subreadall_eio_shard_1: local dir=td/test-erasure-eio 2024-09-16T08:29:15.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:345: TEST_rados_get_with_subreadall_eio_shard_1: local shard_id=1 2024-09-16T08:29:15.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:347: TEST_rados_get_with_subreadall_eio_shard_1: setup_osds 4 2024-09-16T08:29:15.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-16T08:29:15.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-16T08:29:15.072 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-16T08:29:15.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-16T08:29:15.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:29:15.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-16T08:29:15.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:29:15.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:29:15.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T08:29:15.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:29:15.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-16T08:29:15.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:29:15.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:29:15.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:29:15.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:29:15.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-16T08:29:15.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-16T08:29:15.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:29:15.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:29:15.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:29:15.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:29:15.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:29:15.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:29:15.078 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:29:15.078 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:29:15.078 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:29:15.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:29:15.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:29:15.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:29:15.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:29:15.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:29:15.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:29:15.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:29:15.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:29:15.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:29:15.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:29:15.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:29:15.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:29:15.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:29:15.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-16T08:29:15.080 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:29:15.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=50e89e8f-312a-4739-8a9a-85bef53df63a 2024-09-16T08:29:15.080 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 50e89e8f-312a-4739-8a9a-85bef53df63a 2024-09-16T08:29:15.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 50e89e8f-312a-4739-8a9a-85bef53df63a' 2024-09-16T08:29:15.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:29:15.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBb7Odm840/BhAADt/vSvCn7iayKhQlo528ow== 2024-09-16T08:29:15.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBb7Odm840/BhAADt/vSvCn7iayKhQlo528ow=="}' 2024-09-16T08:29:15.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 50e89e8f-312a-4739-8a9a-85bef53df63a -i td/test-erasure-eio/0/new.json 2024-09-16T08:29:15.467 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:29:15.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-16T08:29:15.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBb7Odm840/BhAADt/vSvCn7iayKhQlo528ow== --osd-uuid 50e89e8f-312a-4739-8a9a-85bef53df63a 2024-09-16T08:29:15.511 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:15.509+0000 7fd374423880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:29:15.515 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:15.513+0000 7fd374423880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:29:15.517 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:15.513+0000 7fd374423880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:29:15.518 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:15.517+0000 7fd374423880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T08:29:15.518 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:15.517+0000 7fd374423880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T08:29:15.518 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:15.517+0000 7fd374423880 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-16T08:29:15.617 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:15.613+0000 7fd374423880 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-16T08:29:21.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-16T08:29:21.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:29:21.285 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-16T08:29:21.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T08:29:21.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:29:21.827 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-16T08:29:21.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T08:29:21.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:29:21.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:29:21.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:29:21.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:29:21.864 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:21.861+0000 7f81d73e7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:29:21.867 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:21.865+0000 7f81d73e7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:29:21.869 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:21.865+0000 7f81d73e7880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:29:22.163 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:22.161+0000 7f81d73e7880 -1 Falling back to public interface 2024-09-16T08:29:22.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T08:29:22.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:29:22.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T08:29:22.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:29:22.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:29:22.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:29:22.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:29:22.284 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:29:22.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:29:22.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:29:22.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:29:23.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:29:23.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:29:23.723 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:29:23.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:29:23.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:29:23.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:29:24.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:29:24.355 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:24.353+0000 7f81d73e7880 -1 osd.0 0 log_to_monitors true 2024-09-16T08:29:25.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:29:25.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:29:25.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:29:25.125 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:29:25.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:29:25.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:29:25.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:29:26.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:29:26.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:29:26.673 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:29:26.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:29:26.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:29:26.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:29:27.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:29:28.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:29:28.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:29:28.248 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:29:28.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:29:28.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:29:28.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:29:28.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:29:29.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:29:29.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:29:29.683 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:29:29.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:29:29.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:29:29.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:29:30.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:29:31.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:29:31.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:29:31.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:29:31.215 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:29:31.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:29:31.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:29:31.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:29:32.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:29:32.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:29:32.663 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:29:32.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:29:32.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:29:32.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:29:33.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:29:34.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:29:34.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:29:34.238 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:29:34.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:29:34.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:29:34.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:29:34.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:29:35.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:29:35.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:29:35.692 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:29:35.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:29:35.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:29:35.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:29:36.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:29:37.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:29:37.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:29:37.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:29:37.208 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:29:37.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:29:37.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:29:38.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:29:39.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:29:39.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:29:39.180 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:29:39.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:29:39.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:29:39.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:29:39.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:29:40.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:29:40.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:29:40.855 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:29:40.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:29:40.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:29:40.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:29:41.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:29:42.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:29:42.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:29:42.658 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:29:42.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:29:42.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:29:42.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:29:43.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:29:44.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:29:44.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:29:44.281 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:29:44.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:29:44.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:29:44.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:29:45.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:29:46.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:29:46.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:29:46.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:29:46.287 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:29:46.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:29:46.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:29:46.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:29:47.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:29:47.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:29:47.739 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:29:47.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:29:47.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:29:47.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T08:29:48.230 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2107073470,v1:127.0.0.1:6803/2107073470] [v2:127.0.0.1:6804/2107073470,v1:127.0.0.1:6805/2107073470] exists,up 50e89e8f-312a-4739-8a9a-85bef53df63a 2024-09-16T08:29:48.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:29:48.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:29:48.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:29:48.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:29:48.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-16T08:29:48.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:29:48.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:29:48.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T08:29:48.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:29:48.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-16T08:29:48.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:29:48.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:29:48.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:29:48.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:29:48.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-16T08:29:48.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-16T08:29:48.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:29:48.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:29:48.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:29:48.234 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:29:48.234 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:29:48.234 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:29:48.234 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:29:48.234 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:29:48.235 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:29:48.236 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:29:48.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:29:48.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:29:48.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:29:48.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:29:48.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:29:48.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:29:48.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:29:48.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:29:48.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:29:48.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:29:48.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:29:48.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:29:48.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-16T08:29:48.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:29:48.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=55023e27-47b2-4d10-b8fb-e602922920f8 2024-09-16T08:29:48.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 55023e27-47b2-4d10-b8fb-e602922920f8' 2024-09-16T08:29:48.238 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 55023e27-47b2-4d10-b8fb-e602922920f8 2024-09-16T08:29:48.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:29:48.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB87Odmk5V9DxAAXcvh0vu5mH7qVCqPsLZfAg== 2024-09-16T08:29:48.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB87Odmk5V9DxAAXcvh0vu5mH7qVCqPsLZfAg=="}' 2024-09-16T08:29:48.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 55023e27-47b2-4d10-b8fb-e602922920f8 -i td/test-erasure-eio/1/new.json 2024-09-16T08:29:48.828 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:29:48.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-16T08:29:48.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB87Odmk5V9DxAAXcvh0vu5mH7qVCqPsLZfAg== --osd-uuid 55023e27-47b2-4d10-b8fb-e602922920f8 2024-09-16T08:29:48.873 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:48.868+0000 7fdb3bb65880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:29:48.876 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:48.872+0000 7fdb3bb65880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:29:48.878 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:48.876+0000 7fdb3bb65880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:29:48.879 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:48.876+0000 7fdb3bb65880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T08:29:48.879 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:48.876+0000 7fdb3bb65880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T08:29:48.879 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:48.876+0000 7fdb3bb65880 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-16T08:29:48.958 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:48.952+0000 7fdb3bb65880 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-16T08:29:55.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-16T08:29:55.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:29:55.157 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-16T08:29:55.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T08:29:55.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:29:55.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T08:29:55.731 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-16T08:29:55.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:29:55.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:29:55.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:29:55.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:29:55.765 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:55.760+0000 7fdb322e1880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:29:55.779 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:55.776+0000 7fdb322e1880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:29:55.781 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:55.776+0000 7fdb322e1880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:29:56.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T08:29:56.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:29:56.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T08:29:56.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:29:56.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:29:56.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:29:56.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:29:56.159 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:29:56.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:29:56.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:29:56.340 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:56.336+0000 7fdb322e1880 -1 Falling back to public interface 2024-09-16T08:29:56.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:29:57.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:29:57.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:29:57.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:29:57.652 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:29:57.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:29:57.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:29:58.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:29:58.727 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:29:58.724+0000 7fdb322e1880 -1 osd.1 0 log_to_monitors true 2024-09-16T08:29:59.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:29:59.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:29:59.149 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:29:59.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:29:59.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:29:59.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:29:59.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:00.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:00.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:00.704 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:30:00.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:30:00.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:00.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:30:01.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:02.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:02.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:02.178 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:30:02.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:30:02.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:02.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:30:02.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:03.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:03.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:03.668 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:30:03.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:30:03.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:03.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:30:04.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:05.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:05.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:05.171 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:30:05.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:30:05.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:05.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:30:05.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:06.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:06.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:06.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:30:06.637 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:30:06.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:06.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:30:07.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:08.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:08.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:08.100 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:30:08.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:30:08.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:08.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:30:08.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:09.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:09.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:09.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:30:09.608 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:30:09.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:09.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:30:10.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:11.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:11.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:11.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:30:11.127 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:30:11.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:11.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:30:11.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:12.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:12.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:12.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:30:12.645 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:30:12.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:12.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:30:13.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:14.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:14.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:14.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:30:14.396 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:30:14.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:14.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:30:15.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:16.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:16.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:16.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:30:16.262 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:30:16.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:16.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:30:17.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:18.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:18.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:18.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:30:18.020 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:30:18.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:18.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:30:18.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:19.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:19.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:19.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:30:19.531 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:30:19.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:19.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:30:20.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:21.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:21.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:21.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:30:21.036 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:30:21.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:21.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:30:21.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:22.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:22.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:22.963 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:30:22.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:30:22.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:22.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T08:30:23.577 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/941094874,v1:127.0.0.1:6811/941094874] [v2:127.0.0.1:6812/941094874,v1:127.0.0.1:6813/941094874] exists,up 55023e27-47b2-4d10-b8fb-e602922920f8 2024-09-16T08:30:23.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:30:23.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:30:23.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:30:23.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:30:23.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-16T08:30:23.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:30:23.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:30:23.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T08:30:23.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:30:23.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-16T08:30:23.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:30:23.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:30:23.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:30:23.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:30:23.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-16T08:30:23.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-16T08:30:23.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:30:23.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:30:23.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:30:23.581 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:30:23.581 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:30:23.581 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:30:23.583 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:30:23.583 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:30:23.583 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:30:23.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:30:23.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:30:23.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:30:23.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:30:23.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:30:23.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:30:23.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:30:23.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:30:23.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:30:23.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:30:23.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:30:23.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:30:23.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:30:23.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-16T08:30:23.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:30:23.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1b08d476-9249-4ad2-8be6-f1743836395c 2024-09-16T08:30:23.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 1b08d476-9249-4ad2-8be6-f1743836395c' 2024-09-16T08:30:23.586 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 1b08d476-9249-4ad2-8be6-f1743836395c 2024-09-16T08:30:23.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:30:23.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCf7OdmXFRpJBAAWp7aQcSV6fw59ynioK1Kxg== 2024-09-16T08:30:23.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCf7OdmXFRpJBAAWp7aQcSV6fw59ynioK1Kxg=="}' 2024-09-16T08:30:23.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1b08d476-9249-4ad2-8be6-f1743836395c -i td/test-erasure-eio/2/new.json 2024-09-16T08:30:24.176 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:30:24.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-16T08:30:24.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCf7OdmXFRpJBAAWp7aQcSV6fw59ynioK1Kxg== --osd-uuid 1b08d476-9249-4ad2-8be6-f1743836395c 2024-09-16T08:30:24.229 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:30:24.223+0000 7f73ca1e9880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:30:24.257 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:30:24.251+0000 7f73ca1e9880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:30:24.260 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:30:24.255+0000 7f73ca1e9880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:30:24.261 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:30:24.255+0000 7f73ca1e9880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:30:24.261 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:30:24.255+0000 7f73ca1e9880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:30:24.261 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:30:24.255+0000 7f73ca1e9880 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-16T08:30:24.365 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:30:24.359+0000 7f73ca1e9880 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-16T08:30:30.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-16T08:30:30.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:30:30.413 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-16T08:30:30.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T08:30:30.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:30:30.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T08:30:30.971 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-16T08:30:30.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:30:30.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:30:30.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:30:30.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:30:31.005 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:30:30.999+0000 7f5788d09880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:30:31.037 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:30:31.031+0000 7f5788d09880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:30:31.039 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:30:31.035+0000 7f5788d09880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:30:31.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T08:30:31.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:30:31.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T08:30:31.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:30:31.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:30:31.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:31.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:30:31.501 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:30:31.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:30:31.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:31.595 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:30:31.591+0000 7f5788d09880 -1 Falling back to public interface 2024-09-16T08:30:31.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:32.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:32.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:32.956 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:30:32.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:30:32.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:32.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:30:33.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:34.217 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:30:34.211+0000 7f5788d09880 -1 osd.2 0 log_to_monitors true 2024-09-16T08:30:34.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:34.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:34.390 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:30:34.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:30:34.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:34.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:30:34.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:35.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:35.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:35.920 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:30:35.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:30:35.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:35.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:30:36.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:37.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:37.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:37.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:30:37.371 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:30:37.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:37.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:30:37.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:38.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:38.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:38.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:30:38.783 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:30:38.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:38.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:30:39.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:40.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:40.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:40.358 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:30:40.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:30:40.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:40.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:30:40.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:41.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:41.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:41.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:30:41.810 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:30:41.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:41.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:30:42.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:43.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:43.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:43.283 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:30:43.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:30:43.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:43.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:30:43.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:44.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:44.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:44.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:30:44.732 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:30:44.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:44.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:30:45.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:46.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:46.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:46.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:30:46.163 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:30:46.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:46.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:30:46.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:47.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:47.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:47.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:30:47.602 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:30:47.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:47.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:30:48.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:49.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:49.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:49.247 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:30:49.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:30:49.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:49.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:30:50.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:51.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:51.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:51.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:30:51.078 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:30:51.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:51.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:30:51.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:52.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:52.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:52.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:30:52.879 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:30:52.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:52.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:30:53.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:54.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:54.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:54.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:30:54.640 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:30:54.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:54.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:30:55.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:56.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:56.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:56.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:30:56.528 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:30:56.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:56.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:30:57.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:30:58.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:30:58.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:30:58.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:30:58.020 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:30:58.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:30:58.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:30:58.525 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/744296202,v1:127.0.0.1:6819/744296202] [v2:127.0.0.1:6820/744296202,v1:127.0.0.1:6821/744296202] exists,up 1b08d476-9249-4ad2-8be6-f1743836395c 2024-09-16T08:30:58.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:30:58.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:30:58.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:30:58.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-16T08:30:58.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-16T08:30:58.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-16T08:30:58.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T08:30:58.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T08:30:58.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T08:30:58.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-16T08:30:58.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:30:58.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:30:58.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:30:58.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:30:58.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-16T08:30:58.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-16T08:30:58.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T08:30:58.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T08:30:58.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:30:58.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T08:30:58.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:30:58.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:30:58.530 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:30:58.530 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:30:58.530 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:30:58.530 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:30:58.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:30:58.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:30:58.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T08:30:58.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T08:30:58.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:30:58.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:30:58.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:30:58.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:30:58.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:30:58.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:30:58.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T08:30:58.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T08:30:58.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-16T08:30:58.532 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T08:30:58.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f7a0d149-0308-4def-af1f-71bd4ddd2468 2024-09-16T08:30:58.534 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 f7a0d149-0308-4def-af1f-71bd4ddd2468 2024-09-16T08:30:58.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 f7a0d149-0308-4def-af1f-71bd4ddd2468' 2024-09-16T08:30:58.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T08:30:58.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDC7OdmyXFOIRAA+g9ENm4vAm44H+gpoNCUEg== 2024-09-16T08:30:58.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDC7OdmyXFOIRAA+g9ENm4vAm44H+gpoNCUEg=="}' 2024-09-16T08:30:58.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f7a0d149-0308-4def-af1f-71bd4ddd2468 -i td/test-erasure-eio/3/new.json 2024-09-16T08:30:59.291 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:30:59.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-16T08:30:59.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDC7OdmyXFOIRAA+g9ENm4vAm44H+gpoNCUEg== --osd-uuid f7a0d149-0308-4def-af1f-71bd4ddd2468 2024-09-16T08:30:59.339 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:30:59.335+0000 7fa6d84b4880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:30:59.342 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:30:59.339+0000 7fa6d84b4880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:30:59.345 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:30:59.339+0000 7fa6d84b4880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:30:59.346 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:30:59.343+0000 7fa6d84b4880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:30:59.346 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:30:59.343+0000 7fa6d84b4880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:30:59.346 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:30:59.343+0000 7fa6d84b4880 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-16T08:30:59.467 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:30:59.467+0000 7fa6d84b4880 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-16T08:31:05.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-16T08:31:05.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T08:31:05.537 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-16T08:31:05.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T08:31:05.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T08:31:06.111 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-16T08:31:06.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T08:31:06.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T08:31:06.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T08:31:06.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T08:31:06.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:31:06.145 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:31:06.138+0000 7fefe6dab880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:31:06.167 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:31:06.162+0000 7fefe6dab880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:31:06.169 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:31:06.162+0000 7fefe6dab880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:31:06.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T08:31:06.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:31:06.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T08:31:06.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:31:06.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:31:06.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:31:06.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:31:06.554 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:31:06.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:31:06.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:31:06.728 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:31:06.722+0000 7fefe6dab880 -1 Falling back to public interface 2024-09-16T08:31:07.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:31:08.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:31:08.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:31:08.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:31:08.048 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:31:08.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:31:08.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:31:08.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:31:09.019 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:31:09.014+0000 7fefe6dab880 -1 osd.3 0 log_to_monitors true 2024-09-16T08:31:09.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:31:09.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:31:09.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:31:09.471 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:31:09.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:31:09.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:31:10.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:31:11.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:31:11.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:31:11.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:31:11.176 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:31:11.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:31:11.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:31:11.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:31:12.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:31:12.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:31:12.752 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-16T08:31:12.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T08:31:12.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:31:12.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:31:13.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:31:14.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:31:14.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:31:14.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T08:31:14.173 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-16T08:31:14.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:31:14.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:31:14.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:31:15.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:31:15.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:31:15.610 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-16T08:31:15.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T08:31:15.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:31:15.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:31:16.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:31:17.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:31:17.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:31:17.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T08:31:17.049 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-16T08:31:17.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:31:17.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:31:17.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:31:18.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:31:18.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:31:18.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T08:31:18.545 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-16T08:31:18.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:31:18.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:31:19.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:31:20.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:31:20.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:31:20.098 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-16T08:31:20.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T08:31:20.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:31:20.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:31:20.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:31:21.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:31:21.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:31:21.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T08:31:21.544 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-16T08:31:21.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:31:21.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:31:21.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:31:22.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:31:22.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:31:22.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T08:31:22.984 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-16T08:31:22.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:31:22.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:31:23.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:31:24.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:31:24.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:31:24.993 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-16T08:31:24.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T08:31:24.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:31:24.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:31:25.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:31:26.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:31:26.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:31:26.843 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-16T08:31:26.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T08:31:26.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:31:26.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:31:27.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:31:28.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:31:28.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:31:28.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T08:31:28.800 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-16T08:31:28.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:31:28.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:31:29.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:31:30.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:31:30.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:31:30.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T08:31:30.394 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-16T08:31:30.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:31:30.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:31:31.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:31:32.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:31:32.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:31:32.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T08:31:32.572 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-16T08:31:32.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:31:32.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:31:33.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:31:34.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:31:34.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:31:34.237 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-16T08:31:34.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T08:31:34.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:31:34.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T08:31:34.918 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 29 up_thru 30 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/259920894,v1:127.0.0.1:6827/259920894] [v2:127.0.0.1:6828/259920894,v1:127.0.0.1:6829/259920894] exists,up f7a0d149-0308-4def-af1f-71bd4ddd2468 2024-09-16T08:31:34.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:31:34.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:31:34.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:31:34.920 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-16T08:31:34.920 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T08:31:34.920 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T08:31:34.921 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:31:34.921 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:31:34.921 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:31:34.921 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.0.asok 2024-09-16T08:31:34.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-16T08:31:34.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.0.asok log flush 2024-09-16T08:31:35.013 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-16T08:31:35.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-16T08:31:35.028 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-16T08:29:22.173+0000 7f81d73e7880 0 load: jerasure load: lrc 2024-09-16T08:31:35.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:349: TEST_rados_get_with_subreadall_eio_shard_1: local poolname=pool-jerasure 2024-09-16T08:31:35.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:350: TEST_rados_get_with_subreadall_eio_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2024-09-16T08:31:35.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T08:31:35.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-16T08:31:35.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-16T08:31:35.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-16T08:31:35.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-16T08:31:35.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-16T08:31:35.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-16T08:31:36.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-16T08:31:36.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-16T08:31:36.954 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-16T08:31:36.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T08:31:37.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-16T08:31:37.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:31:37.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:31:37.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:31:37.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:31:37.969 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:31:37.970 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:31:37.970 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:31:37.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:31:37.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:31:37.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:31:38.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:31:38.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:31:38.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:31:38.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:31:38.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:31:38.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:31:38.679 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:31:38.679 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:31:38.679 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:31:38.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:31:38.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:31:38.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:31:38.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803814 2024-09-16T08:31:38.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803814 2024-09-16T08:31:38.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803814' 2024-09-16T08:31:38.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:31:38.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:31:38.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=55834574875 2024-09-16T08:31:38.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 55834574875 2024-09-16T08:31:38.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803814 1-55834574875' 2024-09-16T08:31:38.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:31:38.990 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:31:39.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=81604378644 2024-09-16T08:31:39.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 81604378644 2024-09-16T08:31:39.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803814 1-55834574875 2-81604378644' 2024-09-16T08:31:39.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:31:39.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:31:39.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=124554051601 2024-09-16T08:31:39.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 124554051601 2024-09-16T08:31:39.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803814 1-55834574875 2-81604378644 3-124554051601' 2024-09-16T08:31:39.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:31:39.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803814 2024-09-16T08:31:39.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:31:39.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:31:39.304 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803814 2024-09-16T08:31:39.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:31:39.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803814 2024-09-16T08:31:39.307 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803814 2024-09-16T08:31:39.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803814' 2024-09-16T08:31:39.308 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:31:39.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803814 -lt 25769803814 2024-09-16T08:31:39.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:31:39.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-55834574875 2024-09-16T08:31:39.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:31:39.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:31:39.743 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-55834574875 2024-09-16T08:31:39.743 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:31:39.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=55834574875 2024-09-16T08:31:39.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 55834574875' 2024-09-16T08:31:39.745 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574875 2024-09-16T08:31:39.746 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:31:40.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 55834574875 -lt 55834574875 2024-09-16T08:31:40.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:31:40.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-81604378644 2024-09-16T08:31:40.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:31:40.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:31:40.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-81604378644 2024-09-16T08:31:40.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:31:40.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=81604378644 2024-09-16T08:31:40.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 81604378644' 2024-09-16T08:31:40.196 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 81604378644 2024-09-16T08:31:40.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:31:40.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378642 -lt 81604378644 2024-09-16T08:31:40.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T08:31:41.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T08:31:41.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:31:42.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 81604378644 -lt 81604378644 2024-09-16T08:31:42.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:31:42.124 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-124554051601 2024-09-16T08:31:42.125 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:31:42.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:31:42.128 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-124554051601 2024-09-16T08:31:42.128 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:31:42.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=124554051601 2024-09-16T08:31:42.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 124554051601' 2024-09-16T08:31:42.130 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 124554051601 2024-09-16T08:31:42.131 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:31:42.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 124554051602 -lt 124554051601 2024-09-16T08:31:42.561 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:31:42.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:31:42.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:31:43.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:31:43.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:31:43.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:31:43.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:31:43.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:31:43.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:31:43.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:31:43.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:31:43.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:31:43.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:31:43.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:31:43.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:31:43.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:31:43.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:31:43.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:31:43.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:352: TEST_rados_get_with_subreadall_eio_shard_1: rados_put_get_data eio td/test-erasure-eio 1 recovery 2024-09-16T08:31:43.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-09-16T08:31:43.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-16T08:31:43.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-16T08:31:43.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-16T08:31:43.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2024-09-16T08:31:43.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-16T08:31:43.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg=recovery 2024-09-16T08:31:43.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-16T08:31:43.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-79228-1 2024-09-16T08:31:43.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-79228-1 2024-09-16T08:31:43.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-16T08:31:43.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-16T08:31:43.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-79228-1 2024-09-16T08:31:43.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:31:43.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-16T08:31:43.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:31:43.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-16T08:31:43.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:31:43.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-16T08:31:43.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-16T08:31:43.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-16T08:31:43.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-79228-1 td/test-erasure-eio/ORIGINAL 2024-09-16T08:31:44.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-79228-1 td/test-erasure-eio 1 2024-09-16T08:31:44.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=ec 2024-09-16T08:31:44.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2024-09-16T08:31:44.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2024-09-16T08:31:44.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2024-09-16T08:31:44.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=pool-jerasure 2024-09-16T08:31:44.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2024-09-16T08:31:44.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=obj-eio-79228-1 2024-09-16T08:31:44.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2024-09-16T08:31:44.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/test-erasure-eio 2024-09-16T08:31:44.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2024-09-16T08:31:44.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=1 2024-09-16T08:31:44.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2024-09-16T08:31:44.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:31:44.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds pool-jerasure obj-eio-79228-1 2024-09-16T08:31:44.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:31:44.194 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-79228-1 2024-09-16T08:31:44.194 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-79228-1 2024-09-16T08:31:44.194 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:31:44.672 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:31:44.672 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:31:44.672 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T08:31:44.672 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-16T08:31:44.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2024-09-16T08:31:44.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=1 2024-09-16T08:31:44.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' ec '!=' ec ']' 2024-09-16T08:31:44.674 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/test-erasure-eio/1/type 2024-09-16T08:31:44.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2024-09-16T08:31:44.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-16T08:31:44.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-16T08:31:44.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-16T08:31:44.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-16T08:31:44.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T08:31:44.677 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T08:31:44.677 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-16T08:31:44.678 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T08:31:44.678 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T08:31:44.679 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:31:44.679 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:31:44.679 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:31:44.679 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.1.asok 2024-09-16T08:31:44.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79228/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-16T08:31:44.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T08:31:44.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2024-09-16T08:31:44.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2024-09-16T08:31:44.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.1 2024-09-16T08:31:44.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T08:31:44.785 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T08:31:44.786 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T08:31:44.786 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:31:44.786 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:31:44.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79228/ceph-osd.1.asok 2024-09-16T08:31:44.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2024-09-16T08:31:44.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79228/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-79228-1 1 2024-09-16T08:31:44.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-79228-1 2024-09-16T08:31:44.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-16T08:31:44.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-16T08:31:44.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-79228-1 2024-09-16T08:31:44.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-16T08:31:44.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-16T08:31:44.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-79228-1 td/test-erasure-eio/COPY 2024-09-16T08:31:44.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-16T08:31:44.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-16T08:31:44.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' recovery = recovery ']' 2024-09-16T08:31:44.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: initial_osds=($(get_osds $poolname $objname)) 2024-09-16T08:31:44.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: get_osds pool-jerasure obj-eio-79228-1 2024-09-16T08:31:44.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:31:44.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-79228-1 2024-09-16T08:31:44.942 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-79228-1 2024-09-16T08:31:44.942 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:31:45.380 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:31:45.380 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:31:45.380 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-16T08:31:45.380 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-16T08:31:45.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: local -a initial_osds 2024-09-16T08:31:45.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:171: rados_put_get_data: local last_osd=2 2024-09-16T08:31:45.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:173: rados_put_get_data: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-16T08:31:45.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:31:45.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:31:45.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:31:45.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:31:45.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:31:45.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:31:45.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:174: rados_put_get_data: ceph osd out 2 2024-09-16T08:31:46.967 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 is already out. 2024-09-16T08:31:46.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: get_osds pool-jerasure obj-eio-79228-1 2024-09-16T08:31:46.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: grep '\<2\>' 2024-09-16T08:31:46.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-16T08:31:46.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-79228-1 2024-09-16T08:31:46.978 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T08:31:46.978 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-79228-1 2024-09-16T08:31:47.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-16T08:31:47.551 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:31:47.551 INFO:tasks.workunit.client.0.smithi162.stderr:0' 2024-09-16T08:31:47.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-16T08:31:47.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:176: rados_put_get_data: ceph osd in 2 2024-09-16T08:31:48.509 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 is already in. 2024-09-16T08:31:48.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:177: rados_put_get_data: activate_osd td/test-erasure-eio 2 2024-09-16T08:31:48.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-16T08:31:48.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T08:31:48.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T08:31:48.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T08:31:48.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-16T08:31:48.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-16T08:31:48.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T08:31:48.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T08:31:48.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T08:31:48.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-16T08:31:48.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-16T08:31:48.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T08:31:48.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T08:31:48.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-16T08:31:48.524 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T08:31:48.524 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T08:31:48.524 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T08:31:48.525 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T08:31:48.525 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:31:48.525 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:31:48.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:31:48.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' 2024-09-16T08:31:48.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T08:31:48.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-16T08:31:48.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-16T08:31:48.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T08:31:48.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T08:31:48.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T08:31:48.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T08:31:48.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T08:31:48.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T08:31:48.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-16T08:31:48.527 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-16T08:31:48.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T08:31:48.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0d5c3864-e702-436f-9720-eccb4eb84109 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79228/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T08:31:48.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-16T08:31:48.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T08:31:48.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T08:31:48.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T08:31:48.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T08:31:48.565 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:31:48.562+0000 7fbfa0e7b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:31:48.598 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:31:48.598+0000 7fbfa0e7b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:31:48.600 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:31:48.598+0000 7fbfa0e7b880 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T08:31:49.161 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:31:49.158+0000 7fbfa0e7b880 -1 Falling back to public interface 2024-09-16T08:31:49.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T08:31:49.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T08:31:49.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T08:31:49.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T08:31:49.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T08:31:49.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:31:49.295 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-16T08:31:49.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T08:31:49.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:31:49.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:31:49.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:31:50.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:31:50.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:31:50.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T08:31:50.802 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-16T08:31:50.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:31:50.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:31:51.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:31:52.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:31:52.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:31:52.245 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-16T08:31:52.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T08:31:52.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:31:52.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:31:52.294 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-16T08:31:52.289+0000 7fbfa0e7b880 -1 osd.2 37 log_to_monitors true 2024-09-16T08:31:52.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T08:31:53.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T08:31:53.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T08:31:53.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T08:31:53.721 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-16T08:31:53.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T08:31:53.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T08:31:54.226 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 47 up_thru 19 down_at 38 last_clean_interval [19,37) [v2:127.0.0.1:6818/4261583703,v1:127.0.0.1:6819/4261583703] [v2:127.0.0.1:6820/4261583703,v1:127.0.0.1:6821/4261583703] exists,up 1b08d476-9249-4ad2-8be6-f1743836395c 2024-09-16T08:31:54.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T08:31:54.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T08:31:54.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T08:31:54.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:178: rados_put_get_data: wait_for_clean 2024-09-16T08:31:54.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T08:31:54.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T08:31:54.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T08:31:54.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-16T08:31:54.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T08:31:54.229 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T08:31:54.229 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T08:31:54.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T08:31:54.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T08:31:54.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T08:31:54.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T08:31:54.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T08:31:54.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T08:31:54.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T08:31:54.454 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T08:31:54.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T08:31:54.954 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-16T08:31:54.955 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-16T08:31:54.955 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-16T08:31:54.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T08:31:54.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:31:54.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T08:31:55.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=25769803819 2024-09-16T08:31:55.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 25769803819 2024-09-16T08:31:55.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803819' 2024-09-16T08:31:55.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:31:55.102 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T08:31:55.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=55834574880 2024-09-16T08:31:55.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 55834574880 2024-09-16T08:31:55.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803819 1-55834574880' 2024-09-16T08:31:55.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:31:55.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T08:31:55.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=201863462915 2024-09-16T08:31:55.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 201863462915 2024-09-16T08:31:55.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803819 1-55834574880 2-201863462915' 2024-09-16T08:31:55.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T08:31:55.418 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T08:31:55.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=124554051607 2024-09-16T08:31:55.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 124554051607 2024-09-16T08:31:55.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-25769803819 1-55834574880 2-201863462915 3-124554051607' 2024-09-16T08:31:55.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:31:55.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-25769803819 2024-09-16T08:31:55.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:31:55.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T08:31:55.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-25769803819 2024-09-16T08:31:55.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:31:55.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=25769803819 2024-09-16T08:31:55.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 25769803819' 2024-09-16T08:31:55.578 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803819 2024-09-16T08:31:55.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:31:56.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803817 -lt 25769803819 2024-09-16T08:31:56.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T08:31:57.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T08:31:57.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T08:31:57.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 25769803819 -lt 25769803819 2024-09-16T08:31:57.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:31:57.516 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-55834574880 2024-09-16T08:31:57.516 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:31:57.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T08:31:57.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-55834574880 2024-09-16T08:31:57.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:31:57.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=55834574880 2024-09-16T08:31:57.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 55834574880' 2024-09-16T08:31:57.522 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574880 2024-09-16T08:31:57.522 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T08:31:57.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 55834574880 -lt 55834574880 2024-09-16T08:31:57.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:31:57.972 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-201863462915 2024-09-16T08:31:57.972 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:31:57.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T08:31:57.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-201863462915 2024-09-16T08:31:57.976 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:31:57.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=201863462915 2024-09-16T08:31:57.978 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 201863462915 2024-09-16T08:31:57.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 201863462915' 2024-09-16T08:31:57.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T08:31:58.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 201863462915 -lt 201863462915 2024-09-16T08:31:58.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T08:31:58.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-124554051607 2024-09-16T08:31:58.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T08:31:58.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T08:31:58.422 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-124554051607 2024-09-16T08:31:58.422 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T08:31:58.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=124554051607 2024-09-16T08:31:58.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 124554051607' 2024-09-16T08:31:58.425 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 124554051607 2024-09-16T08:31:58.425 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T08:31:58.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 124554051607 -lt 124554051607 2024-09-16T08:31:58.830 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T08:31:58.831 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:31:58.831 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:31:59.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-16T08:31:59.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T08:31:59.398 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T08:31:59.398 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T08:31:59.399 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T08:31:59.399 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T08:31:59.399 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T08:31:59.399 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T08:31:59.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-16T08:31:59.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T08:31:59.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T08:31:59.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T08:32:00.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-16T08:32:00.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T08:32:00.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T08:32:00.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:354: TEST_rados_get_with_subreadall_eio_shard_1: delete_erasure_coded_pool pool-jerasure 2024-09-16T08:32:00.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-16T08:32:00.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-16T08:32:01.001 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-16T08:32:01.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-16T08:32:01.829 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-16T08:32:01.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-16T08:32:01.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T08:32:01.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T08:32:01.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T08:32:01.842 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:32:01.842 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:32:01.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:32:01.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:32:01.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:32:01.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:32:01.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T08:32:01.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T08:32:01.963 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T08:32:01.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T08:32:01.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T08:32:01.966 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T08:32:01.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:32:01.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T08:32:01.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T08:32:01.969 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:32:01.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:32:01.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T08:32:01.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T08:32:01.998 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T08:32:01.998 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:32:01.998 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:32:01.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T08:32:02.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T08:32:02.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T08:32:02.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T08:32:02.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/test-erasure-eio 0 2024-09-16T08:32:02.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-16T08:32:02.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T08:32:02.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-16T08:32:02.002 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T08:32:02.002 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T08:32:02.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T08:32:02.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T08:32:02.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T08:32:02.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T08:32:02.008 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T08:32:02.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T08:32:02.010 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T08:32:02.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T08:32:02.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T08:32:02.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T08:32:02.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:32:02.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T08:32:02.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T08:32:02.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:32:02.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:32:02.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T08:32:02.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-16T08:32:02.021 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T08:32:02.021 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T08:32:02.021 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79228 2024-09-16T08:32:02.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79228 2024-09-16T08:32:02.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T08:32:02.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T08:32:02.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T08:32:02.025 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T08:32:02.025 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T08:32:02.081 INFO:tasks.workunit:Stopping ['erasure-code'] on client.0... 2024-09-16T08:32:02.082 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-16T08:32:02.360 DEBUG:teuthology.parallel:result is None 2024-09-16T08:32:02.360 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T08:32:02.375 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T08:32:02.375 DEBUG:teuthology.orchestra.run.smithi162:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T08:32:02.429 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T08:32:02.430 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T08:32:02.439 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T08:32:02.439 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T08:32:02.479 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T08:32:02.487 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T08:32:02.487 DEBUG:teuthology.orchestra.run.smithi162:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T08:32:02.520 INFO:teuthology.orchestra.run.smithi162.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T08:32:02.520 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================== 2024-09-16T08:32:02.520 INFO:teuthology.orchestra.run.smithi162.stdout:*hv01.front.sepi 67.205.162.81 3 u 228 256 377 0.135 -1.063 0.271 2024-09-16T08:32:02.520 INFO:teuthology.orchestra.run.smithi162.stdout:+hv02.front.sepi 96.245.170.99 3 u 103 256 377 0.169 3.385 0.275 2024-09-16T08:32:02.521 INFO:teuthology.orchestra.run.smithi162.stdout:+hv03.front.sepi 74.6.168.72 3 u 228 256 377 0.166 -1.149 0.249 2024-09-16T08:32:02.521 INFO:teuthology.orchestra.run.smithi162.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2024-09-16T08:32:02.521 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T08:32:02.529 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T08:32:02.530 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T08:32:02.580 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T08:32:02.634 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T08:32:02.680 INFO:teuthology.task.internal:Duration was 5230.388169 seconds 2024-09-16T08:32:02.681 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T08:32:02.730 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T08:32:02.730 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T08:32:02.781 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T08:32:02.781 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi162.front.sepia.ceph.com 2024-09-16T08:32:02.782 DEBUG:teuthology.orchestra.run.smithi162:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T08:32:02.793 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T08:32:02.794 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T08:32:02.885 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T08:32:02.885 DEBUG:teuthology.orchestra.run.smithi162:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T08:32:02.952 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T08:32:02.962 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T08:32:02.962 DEBUG:teuthology.orchestra.run.smithi162:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T08:32:02.975 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T08:32:02.988 DEBUG:teuthology.orchestra.run.smithi162:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:32:03.032 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = core 2024-09-16T08:32:03.047 DEBUG:teuthology.orchestra.run.smithi162:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T08:32:03.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T08:32:03.052 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T08:32:03.064 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T08:32:03.066 DEBUG:teuthology.misc:Transferring archived files from smithi162:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906617/remote/smithi162 2024-09-16T08:32:03.067 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T08:32:03.094 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T08:32:03.094 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T08:32:03.135 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T08:32:03.143 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T08:32:03.144 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T08:32:03.172 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T08:32:03.173 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T08:32:03.180 INFO:teuthology.orchestra.run.smithi162.stdout: 924119 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 08:32 /home/ubuntu/cephtest 2024-09-16T08:32:03.182 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T08:32:03.223 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T08:32:03.325 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_20.04} workloads/erasure-code} duration: 5230.388169050217 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T08:32:03.325 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T08:32:03.403 INFO:teuthology.run:pass